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Preface 


This book was written as a textbook for an undergraduate, senior-level course 
on random processes for Statistics majors. The presentation is meant to be 
light yet sufficiently mathematical, with good, interesting, scientific appli- 
cations. More advanced topics, such as sigma algebras, martingales, general 
renewal processes, Levy processes, and stochastic calculus, are deliberately 
avoided. The knowledge of statistics is limited to a linear regression, goodness 
of fit test, and point estimation. 


There are nine chapters in this book, covering Markov chain, random 
walk, Poisson processes (homogeneous, nonhomogeneous, compound, and con- 
ditional), birth-and-death process, branching process, and Brownian motion. 
Each chapter gives just enough theory in the form of definitions, propositions, 
remarks, and examples, followed by a section on simulation of trajectories. 
Finally, applications of processes are presented. At the end of each chapter, a 
collection of exercises is included. Some of the exercises are theoretical whereas 
some others require calculation and simulation. 


The R software is used throughout. Complete codes and relevant outputs 
are given in the text. The website that accompanies this book 


https://home. csulb. edu/~ okoroste/stochprocesses. html 


contains complete R codes for all examples and applications, and the data 
sets in .csv format that are used in applications and/or required for certain 
exercises. A complete solutions manual is also available to instructors upon 
request at https: //www.routledge.com/9781032153735. 


Last but not least, I would like to thank John Peach who helped me to sim- 
ulate a trajectory of a branching process by means of self-referencing functions. 
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Stochastic Process, Discrete-time Markov 
Chain 


1.1 Definition of Stochastic Process 


A stochastic process {X(t), t € T} is a collection of random variables indexed 
by a parameter ¢t that belongs to a set T. The parameter t is often referred to 
as time, and the value X(t) is the state of the process at time t. The set T is 
called the index set of the process. 


The state space S of a stochastic process is the set of all possible values of 
X(t), for any t € T. 


If T is a countable set, the stochastic process is termed a discrete-time pro- 
cess (or, simply, a discrete process). Otherwise, it is called a continuous-time 
process. 


TT 


1.2 Discrete-time Markov Chain 


A discrete-time Markov chain’ is a discrete-time stochastic process {X,, n = 
0, 1,2, etc.} which state space S is finite or countably infinite and such that 


P(Xnai =7|Xo = to, X1 =H, ..., Xn = 1) = P(Xnsr =| Xn = 1) = Py, 
(1.1) 


that is, the conditional probability of the process being in state 7 at time n+1 
given all the previous states depends only on the last-known position (state 


lIntroduced in Markov, A. A. (1913). “An example of statistical investigation of the text 
Eugene Onegin concerning the connection of samples in chains.” (In Russian.) Bulletin of 
the Imperial Academy of Sciences of St. Petersburg, 7(3): 153 — 162. 
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i at time n). This property is called the Markovian property (or the Markov 
property). The probability P,; is called the one-step transition probability of 
the Markov chain. Note that it is constant for given states 7 and 7. 


In Markov chains, one-step transition probabilities are typically aggregated 
into a one-step transition probability matrix 


Poo Po. Poa 

Pio Pir Piz 
P= ee 

Pio Pa P,; 


EXAMPLE 1.1. Consider a Markov chain with the state space S = {1,2,3} 
and transition probability matrix 


Is 2 Oe 23 
170.7 0.1 0.2 

P= 2 [ac 0.6 oa) 
3 105 02 0.3 


(a) Starting in state 1, the Markov chain returns to it in one step with proba- 
bility 0.7, or transitions to state 2 with probability 0.1, or transitions to state 
3 with probability 0.2. From state 2, the chain cannot transition to state 1 in 
a single step since the probability of this event is 0. It can, however, return to 
state 2 or move to state 3 with probabilities 0.6 and 0.4, respectively. If the 
chain is in state 3, it will transition to states 1 or 2 with respective probabil- 
ities 0.5 and 0.2 or will loop back to state 3 with a probability of 0.3. Note 
that since the chain must transition to some state, the probabilities in each 
row necessarily sum up to 1. 


(b) Conditional probabilities can be computed using the Markov property. For 
example, P(X3 1 | Xo 1,X1 2,X2 3) P(X3 1 | Xo 3) P31 
0.5. 


(c) Joint probabilities can be computed by conditioning and using the Markov 
property. For example, we want to compute the probability that a Markov 
chain starts in state 1 at time 0, then transitions into state 2, and then into 
state 3. We obtain P(Xo 1,X1 2,X2 3) P(X2 3| Xo 1,X, 

2) P(Xo = 1, X1 = 2) = P(Xp = 3] X1 = 2) P(X = 2| Xo = 1)P(Xo0 = 1) = 
Po3 + Pig - P(X = 1) = (0.4)(0.1)(1) = 0.04. 
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a Eee 


1.3. Chapman-Kolmogorov Equations 


Consider a Markov chain with finite or countably infinite state space S = 
{0,1,2,...}. For fixed states 7 and j, the n-step transition probability Pj; is 
the probability that a process that is currently in state z will be in state 7 
after n transitions, that is, for any time m > 0, Pj} = P(Xmtn = j|Xm = 71). 


Denote the n-step transition probability matrix by P™. As a special case with 
n=1,PY =P. 


Next, we will show that P(”) = P”, which indicates that finding the n-step 
transition probability matrix is tantamount to multiplying the one-step tran- 
sition probability matrix by itself n times. 


The proof is based on the Chapman-Kolmogorov equations which assert that 
for all positive integers n and m, P™+™ = P™ . P(™), or, equivalently, 
prim oe gay oe Py, for any states 7 and j. Indeed, by the definition of 
conditional probability, 


10 Re Te ee 
P(Xo =?) 


Next, we fix state k in which trajectory of the Markov chain is located after 
n transitions and sum up the probabilities with respect to k, obtaining: 


PI” = P(Xntm = j|Xo = 1) = 


PX et 9 Sk, Ng) 
prrm — pee Sai a Ale ie A 
Sy 


which by the definition of conditional probability is equal to 


25 P(Xnim = 5|Xn =k, Xo = i)P(Xn = k, Xo = 1) 
> P(Xo = 1) j 


=o P(X Sg hn Sk Xo SDP Xa = kX = 0- 


Finally, applying the Markov property, we can omit from the history all but 
the latest known state that we condition on, and deduce that 


Pete Se Pain =) = OP, Sh een = YP eR: 
k=0 k=0 
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This completes the proof. Now, as a corollary of the Chapman-Kolmogorov 
equations, we will show that P(”) = P”. Applying the method of mathemat- 
ical induction, we first check that the statement is true for small values of 
n: P@) = PG+) = p®).pQ) = p.P = P?, and P®) = P®).P = P®. 
Assuming further that the statement holds for n — 1, we prove it for n. We 
write, P™) =Pp@-).p—=p!1.p=p”., 


EXAMPLE 1.2. In Example 1.1, we considered a Markov chain with the one- 
step transition probability matrix 


0.7 O.1 0.2 
P=/0.0 0.6 0.4 
0.5 0.2 0.3 


(a) Suppose we would like to find a three-step transition probability matrix. 
We can compute 


OF Of. 02) ‘fOr OL 62) [or 02 02 
P®) =P®= 10.0 06 0.4|-10.0 06 0.4] -10.0 0.6 0.4 
0.5 0.2 0.3] 105 02 03] 10.5 02 0.3 


0.59 0.17 0.24 0.7 0.1 0.2 0.533 0.209 0.258 
= |0.20 0.44 0.36] -|0.0 0.6 0.4) = |0.320 0.356 0.324 
0.50 0.23 0.27 0.5 0.2 0.3 0.485 0.242 0.273 


(b) Now we can compute probabilities that require the knowledge of the three- 
step transition probability matrix. For instance, if P(Xp = 1) = 1, we can 
calculate P(Xp = 1,X1 = 2,Xa = 3) = P(X = 3| Xo =1,X1=]DPK(M = 
2| Xo = 1)P(X = 1) = P(X4 = 3| Xy = 2) P(X, = 2| Xp = 1)P(Xp = 1) = 
P2) . Pio -P(Xo = 1) = (0.324)(0.1)(1) = 0.0324. 


Further, suppose we are given the distribution of the initial state p? = P(Xo = 
i). Conditioning on this distribution, we can obtain the unconditional distri- 
bution of the state at time n as 


pP = P(Xn = j) = D> P(Xn = j|Xo = i)P(Xo =i) = DY PH p?. 
i=0 
In the matrix form, this equation becomes 
(of.03,---) = (1, pg,-.-) P™. 


EXAMPLE 1.3. Suppose that in Example 1.1 the initial states are equiprob- 
able, that is, py) = po = p3 = 1/3. Then, after three transitions, the 
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probability that the chain ends up in state 1 is P(X3 = 1) = 4 PS ) p= 
(0.533) (1/3) + (0.320) (1/3) + (0.485) (1/3) = 0.446. Note that in these calcula- 
tions, the three-step transition probabilities come from the first column of the 
three-step transition probability matrix obtained in Example 1.2. Likewise, 
the probability that after three steps the chain will be in state 2 can be found 
as P(X3 = 2) = 73_, Pp; = (0.209)(1/3) + (0.356) (1/3) + (0.242)(1/3) = 
0.269. The unconditional probability of state 3 may be obtained by similar 
calculations P(X3 = 3) = (0.258)(1/3) + (0.324)(1/3) + (0.273)(1/3) = 0.285, 
or by subtraction from 1, P(X3 = 3) = 1 — 0.4646 — 0.269 = 0.285. 


These calculations may be written in the matrix form as follows: 


0.533 0.209 0.258 
(1/3, 1/3, 1/3) }0.320 0.356 0.324] = (0.446, 0.269, 0.285). 
0.485 0.242 0.273 


1.4 Classification of States 


Consider a discrete-time Markov chain {X,, n > 0}. We say that starting in 
state 2, the chain ever enters state j if for some n > 0, Xp = 7, that is, 


co 
{chain ever enters state j|it starts in state i} = U (Kea G | koa ay. 


n=0 


A state 7 is called accessible from state i if, starting in 7, the chain will ever 
enter state 7 with a positive probability. To denote that state 7 is accessible 
from state i, we write i —> 7. 


Further, if two states are accessible from each other, we say that they com- 
municate, and denote it as i 4 7. Communication property is an equivalence 
relation. Indeed, (i) it is reflexive since any state communicates with itself in 
0 steps, (ii) it is symmetric by definition: if state i communicates with state 
j, then state 7 communicates with state i, and (iii) it is transitive because if 
state 7 communicates with state 7, and state 7 communicates with state k, 
then state 7 communicates with state k. 


Now, since communication is an equivalence relation, it means that communi- 
cation is a class property: all states that communicate with each other belong 
to the same class, and the state space of a Markov chain may be partitioned 
into classes. If there is only one class, the chain is termed irreducible. 


Next, a state is called recurrent if with probability 1 the chain ever reenters 
that state. Otherwise, the state is called transient. Any state is either recurrent 
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or transient. Furthermore, at least one state of a finite-state Markov chain 
must be recurrent because if all states are transient, after the chain leaves 
all the states it has no state to go to. Similar to the communication relation, 
recurrence and transience are class properties, and the entire class is called 
recurrent or transient. 


In addition, a state is called absorbing if the chain cannot leave it once it 
enters it. An absorbing Markov chain has at least one absorbing state. A state 
is termed reflecting if once the chain leaves it, it cannot return to it. 


Finally, the period d of a state 7 is the number such that, starting in 2, the 
chain can return to 7 only in the number of steps that are multiples of d. A 
state with period d= 1 is called aperiodic. Periodicity is a class property. 


For a reflecting state, the period is infinite, since the chain never comes back 
to this state. Absorbing states necessarily have loops and thus are aperiodic 
states. 


EXAMPLE 1.4. Referring back to Example 1.1, we first draw a diagram of the 
Markov chain. The R code that produces the diagram will be displayed in 
Example 1.8. 


As seen in the diagram, state 2 can be reached from state 1 directly, and states 
2 and 3 are directly accessible from each other. Also, state 1 is accessible from 
state 2 through state 3. Therefore, all three states communicate, and thus the 
chain has a single class and is irreducible. Since there is only one class, it must 
be recurrent. Also, the chain has loops for every state, thus it can return to 
every state in one step and is aperiodic. 


EXAMPLE 1.5. Consider a Markov chain with the state space {1,2,3,4,5,6} 
and the one-step transition probability matrix 


Limiting Probabilities 7 


1 2 3.4 5 6 
1 03 07 0 0 0 0 
2 1 0 00 0 0 
3 05 0 0 0 O 0.5 
4 0 0 06 0 O 04 
5 0 O O 0 O01 0.9 
6 0 O O 0 0.7 0.3 


The diagram for this chain is given below. The corresponding R code will be 
presented in Example 1.9. 


We can see that states 3 and 4 are reflecting and therefore transient. The chain 
will transition out of both states and will enter the recurrent class {1,2} or 
{5,6}. States 3 and 4 have infinite periods whereas both recurrent classes are 
aperiodic due to the existence of the loops. 


rr 


1.5 Limiting Probabilities 


In a Markov chain, the limiting probability 7; = limn—oo Py doesn’t depend 
on the initial state 7 and can be interpreted as the long-run proportion of time 
that the Markov chain spends in state 7. Limiting probabilities are also termed 
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the limiting distribution or stationary distribution or steady-state distribution. 
If the stationary distribution exists, it satisfies the system of equations: 


15 Peas (1.2) 
i=0 


To see this, we condition on the state at time n and write P(X,41 = j) = 
Se PO S49 Ae POG SS) = 9c 5 Pa a Letting i oo, 
and assuming that we can justify exchanging the limit and the summation 
sign, we get (1.2). 


If we combine the limiting probabilities into a row vector 7 = (71,72,...), 
then the system of equations (1.2) has the matrix form 7 = 7 - P where P 
is the one-step transition probability matrix. Since the rows of the transition 
probability matrix add up to 1, the equations in the system are linearly depen- 
dent. Nonetheless, we can find all limiting probabilities if we take into account 
the fact that they must sum up to 1: 7, +-72+-:-=1. 


A Markov chain that has a unique stationary distribution is referred to as 
ergodic. 


EXAMPLE 1.6. The stationary distribution for the Markov chain considered 
in Examples 1.1 — 1.4 solves 


0.7 O.1 0.2 
(771,772,773) = (71, 72,73) |0.0 0.6 0.4], and m +72+73 =1. 
0.5 0.2 0.3 
Or, equivalently, 
7 = 0.77% + 0.573 73 = 0.671 
Tg = 0.17, + 0.672 + 0.273 , or <4 2 = 0.5571 
TM +72+73 = 1, 2.157, = 1, 


resulting in the solution 7, = 0.4651, 72 = 0.2558, and 73 = 0.2791. It means 
that in a long run, the chain spends roughly 46.5% of the time in state 1, 
25.6% of the time in state 2, and 27.9% of the time in state 3. Since the 
stationary distribution is unique, it is an ergodic chain. 


EXAMPLE 1.7. As will be demonstrated in Example 1.9, for the Markov chain 
in Example 1.5, the system of equations (1.2) has two solutions 


(0.5882, 0.4118, 0,0,0,0) and (0,0, 0,0, 0.4375, 0.5625). 


This happens because the chain contains two recurrent classes, {1,2} and 
{5,6}. Since there are two solutions, the chain is non-ergodic. Denote these 
two vectors by 7, and 72, respectively. Then any linear combination of the 
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form am, + (1 —a@)m2 where 0 < a < 1 is a solution, and therefore, there ac- 
tually exist infinitely many solutions. Any such solution is called an invariant 
measure. Neither is considered to be the stationary distribution. 


a 


1.6 Computations in R 


To mimic in R the analysis done in examples in the previous section, three 
packages are required: “diagram” to plot the diagram, “expm” to exponenti- 
ate matrices, and “markovchain” to determine recurrent and transient classes, 
absorbing states, and the steady-state distribution. The script is as follows: 


install.packages ("diagram") 
install.packages("expm") 
install. packages ("markovchain") 


e First, an n x n transition probability matrix should be specified: 


tm.name<- matrix(c(pi1.value, pio.value,...,Pnn.value), nrow=n.value, 
ncol=n.value, byrow=TRUE) 


e To plot the diagram with the arrows pointing in the correct direction, the 
transition matrix must be transposed. This can be done by typing 


tr.tm.name<- t(tm.name) 


The diagram is drawn using plotmat function. The graph depicts state names 
inside boxes and directed lines connecting the boxes labeled by the correspond- 
ing transition probabilities. The syntax is: 


library (diagram) 
plotmat (tr.tm.name, <arguments>) 


The arguments in the above function are: 


© pos, a vector specifying the number of boxes in each row in the diagram. For 
instance, in a three-state chain, pos=c(1,2) means that state 1 is depicted 
in the top row, and states 2 and 3 are in the next row. By default, boxes are 
plotted in a circle. 

o name=c("statel.name", "state2.name", ... ), the list of state names. By 
default, natural numbers are used. 

o arr.col, color of inside of all arrowheads (excluding the contours). Black 
by default. 
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o arr.1col, color of all arrow lines. Black by default. 

o arr.length, length of all arrows. 

o arr.pos, relative position of arrowheads on lines (excluding loops), a value 
between 0 and 1. By default, arrows are positioned in the middle. 

o arr.type, type of arrowhead, some options are curved, triangle, or 
simple. 

o arr.width, width of all arrows. 

© box. cex, size of labels in boxes (i.e., state names). The magnitude is relative 
to the default value of character expansion. 

© box.col, color of inside of all boxes (excluding contours). By default, the 
color is white. 

© box.1col, color of contours of all boxes. Black by default. 

° box.1wd, width of contours of all boxes. 

© box. prop, ratio of length over width of all boxes. The ratio is equal to 1 by 
default. 

o box.size, size of all boxes. 

© box.type, type of all boxes, some options are rect, ellipse, round (a 
rectangle with rounded edges), circle, diamond, hexa (a hexagonal shape). 
o cex.txt, size of labels next to arrows (i.e., values of respective probabili- 
ties). 

° 1col, color of all lines, contours of all arrows, and contours of all boxes. 
Black by default. 

o 1wd, width of all arrow lines, excluding loops. 

o self .cex, size of all loops (also termed “self-arrows”). 

o self .arrpos, angle in radians of arrow positions on all loops relative to the 
a-axis. 

o self .1wd, width of all loops. 

o self.shiftx, shift of all loops along the z-axis. 

o self.shifty shift of all loops along the y-axis. 

o txt.col, color of labels in boxes (i.e., state names). 


e To compute an n-step transition probability matrix for a specific value of n, 
use the following code: 


library (expm) 
nstepmatrix.name<- tm.name% A %n.value 


e Given the initial distribution, the lines below calculate the unconditional 
distribution after n steps. 


init.p.name <- c(pl.init.value, p2.init.value,...) 
uncond.dist.name<- init.p.nameh*hnstepmatriz.name 


e To determine recurrent and transient classes, absorbing states, and the lim- 
iting distribution, a discrete-time Markov chain must be created as an object. 
It can be done as follows: 
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library (markovchain) 
dtmc.name<- new("markovchain", transitionMatrix=tm.name, 
states=c("statel.name", "state2.name",...)) 


Then state characteristics may be obtained as 


recurrentClasses (dtmc.name) 
transientClasses(dtmc.name) 
absorbingStates (ditmc.name) 
steadyStates (dtmc.name) 


Note that R doesn’t identify reflecting states. 


In R, one can compute the period of an irreducible (a single-class) Markov 
chain only. The syntax is period (dtmc.name). 


EXAMPLE 1.8. The results obtained for the Markov chain from Examples 1.1- 
1.4 and 1.6 can be produced in R as presented below. 


#specifying transition probability matrix 
tm<- matrix(c(0.7, 0.1, 0.2, 0.0, 0.6, 0.4, 0.5, 0.2, 0.3), 
nrow=3, ncol=3, byrow=TRUE) 


#transposing transition probability matrix 
tm.tr<- t(tm) 


#plotting diagram 

library (diagram) 

plotmat(tm.tr, pos=c(1,2), arr.length=0.3, arr.width=0.1, 
box.col="light blue", box.lwd=1, box.prop=0.5, box.size=0.12, 
ere yiie= CGamieglls”,, Coe wee=OS, lbncil, Selle. cerx=0),6), 
self.shiftx=0.17, self.shifty=-0.01) 
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#computing three-step transition probability matrix 
library (expm) 
print (tm3<- tm% A % 3) 


[,1] [,2] [,3] 
[1,] 0.533 0.209 0.258 
[2,] 0.320 0.356 0.324 
[3,] 0.485 0.242 0.273 


#computing unconditional distribution after three steps 
init.p<- c(1/3, 1/3, 1/3) 
init .p% * %tm3 


[,1] (,2] [,3] 
[1,] 0.446 0.269 0.285 


#creating Markov chain object 
library (markovchain) 
mc<- new("markovchain", transitionMatrix=tm, states=c("1", 
Ww "3")) 
? 


#computing Markov chain characteristics 
recurrentClasses (mc) 


Mp 

transientClasses (mc) 
list () 

absorbingStates (mc) 
character (0) 


period(mc) 


round(steadyStates(mc), digits=4) 


1 2 3 
0.4651 0.2558 0.2791 
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EXAMPLE 1.9. Consider the Markov chain from Examples 1.5 and 1.7. We 
run the following R code to produce the diagram, invariant vectors, and to 
verify the state classification. 


#specifying transition probability matrix 
tm<- matrix(c(0.3,0.7,0,0,0,0,1,0,0,0,0,0,0.5 
0,0,0.6,0,0,0.4,0,0,0,0,0.1,0.9,0,0,0,0,0.7,0 
ncol=6, byrow=TRUE) 


50 50,0,0,0.5, 
.3), nrow=6, 


#transposing transition probability matrix 
tm.tr<- t(tm) 


#plotting diagram 

library (diagram) 

plotmat(tm.tr, arr.length=0.3, arr.width=0.1, box.col="light 
blue", box.lwd=1, box.prop=0.5, box.size=0.09, 
box.type="circle", cex.txt=0.8, lwd=1, self.cex=0.3, 
self.arrpos=0.3, self.shiftx=0.09, self.shifty=-0.05) 


#creating Markov chain object 

library (markovchain) 

mc<- new("markovchain", transitionMatrix=tm, states=c("1", 
(ey SU wa Us ul "6")) 


#computing Markov chain characteristics 
recurrentClasses (mc) 


qe wou 


wou me 
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transientClasses (mc) 
wu 


"qu 


#finding periods of irreducible Markov chains 

tm12.ir<- matrix(c(0.3,0.7,1,0), nrow=2, ncol=2, byrow=TRUE) 
mci2.ir<- new("markovchain", transitionMatrix=tm12.ir, 
states=c("1","2")) 

period(mc12.ir) 


tm56.ir<- matrix(c(0.1,0.9,0.7,0.3), nrow=2, ncol=2, 
byrow=TRUE) 

mc56.ir<- new("markovchain", transitionMatrix=tm56.ir, 
states=c("5","6")) 

period(mc56. ir) 


#finding steady-state distribution 
round(steadyStates(mc), digits=4) 


1 2 3 4 5 6 
0.0000 0.0000 0 O 0.4375 0.5625 
0.5882 0.4118 O O 0.0000 0.0000 


rr 


1.7 Simulations in R 


In R, to simulate simultaneously ntraj.value trajectories of a k-state Markov 
chain over nsteps.name time-steps, the function rmarkovchain() in the li- 
brary markovchain can be used. It keeps track of the step number and the 
Markov chain state at that step. Each new state is chosen according to a 
multinomial probability distribution with the probability mass function 


P Sige »>Uk,P1,°°° , Pk) =P(X =2%1,°"° Xk = Lk) 


! 
n! 
=e wees p,*, where 21 +---+2, =n. 
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Given the initial vector of probabilities (p},...,p;), the multivariate proba- 
bility distribution is found recursively: 


(BPs 2 DOr Be spe Ps n= 2,3,..., (1.3) 
where P denotes the one-step transition probability matrix of the Markov 
chain. 


Assuming that the Markov chain object mc.name has already been created, 
the syntax is as follows: 


#specifying total number of trajectories 
ntraj.name<- ntraj.value 


#specifying total number of steps 
nsteps.name<- nsteps.value 


#specifying initial probability 
init.prob.name<- c(pl.value, p2.value, ..., pk.value) 


#specifying matrix containing states 
states.matriz.name<- matrix(NA, nrow=nsteps.name, ncol=ntraj.name) 


#specifying seed 
set .seed (value) 


#generating initial state 
init.state.name<- sample(1:k, 1, prob=init.prob.name) 


#simulating states 

for (i in 1:ntraj.name) 

states.matriz.name[,i]<- rmarkovchain(n=nsteps.name-1, 
object=mc.name, 

t0=init.state.name, include.t0O=TRUE) 


e The value of the seed tells R where to start reading off in the table of random 
digits. This is done for the reproducibility of results. If the code is run again, 
it will produce the same trajectories. 


e The function sample(1:k, 1, prob=init.prob.name) draws one state from 
among the k states and the sampling is done according to the multinomial 
distribution with the probability vector init.prob.name. 
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e The final product of the above code is a matrix with dimensions nsteps.value 
by ntraj.value with columns containing state names for individual trajectories. 


As an alternative to using the built-in function rmarkovchain(), one can cre- 
ate a user-defined function that simulates trajectories. The syntax is 


function.name<- function(tm.name, init.prob.name, nsteps.name) { 
states.name<- numeric (nsteps.name) 
states.name[1]<- sample(1:k, 1, prob=init.prob.name) 


for(t in 2:nsteps.name) { 
prob.name<- tm.name[states.name[t-1] ,] 
states.name[t]<- sample(1:k, 1, prob=prob.name) 
} 
return (states.name) 


} 


set.seed (value) 

states.matriz.name<- matrix(NA, nrow=nsteps.name, ncol=ntraj.name) 
for (j in 1:ntraj.name) 

states.matriz.name<- function.name(tm.name, init.prob.name, 
nsteps.name) 


Finally, the ntraj.value trajectories can be overlayed on a single graph by 
means of the matplot() function with the following syntax: 


#plotting simulated trajectories 
matplot (states.matriz.name, <arguments>) 


where the arguments are: type="1" to connect the dots, 1ty=1 to draw a solid 
line, Lwd=2 to make the lines appear thicker, col=a:b to choose a sequen- 
tial subset a to b of ntraj.name colors from this cyclic list {1=black, 2=red, 
3=green, 4=dark blue, 5=sky blue, 6=pink, 7=yellow, 8=gray, 9=black, 
10=red, 11=green, etc.}, xlim=c(z.lower.value, x.upper.value) to limit the 
a-axis, if needed, ylim=c(y.lower.value, y.upper.value) to limit the y-axis, if 
needed, xlab="step.name" and ylab="state.name" to display step.name on 
the z-axis and state.name on the y-axis. Colors can also be specified as an ex- 
plicit list: col=c("red", "green", "blue"), for example. To add grid lines 
to the coordinate system on the plot, the argument panel.first=grid() is 
used. 


To change tick marks on either axis and to modify labels for ticks, one can re- 
move axes by including in matplot () the arguments xaxt="n" and yaxt="n", 
and then specify the new axes as axis(side=number, at=range for ticks), 
where side=1 is for z-axis, and side=2 is for y-axis. 
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To enhance the clarity of what states the depicted trajectories are at, dots can 
be added at every step. To do so, use points(z values, y values, pch=16), 
where the argument pch=16 identifies the plotting character for the points as 
a filled circle. 


Additionally, to visualize convergence of unconditional probabilities p,,, com- 
puted recursively in accordance with (1.3), the following R syntax can be 
implemented. We assume that there are k states in the Markov chain, and the 
transition probability matrix has been defined as tm.name object. 


#specifying total number of steps 
nsteps.name<- nsteps.value 


#specifying matrix containing probabilities 
probs.matriz.name<- matrix(NA, nrow=nsteps.name, ncol=k) 


#specifying initial probability 
init.prob.name<- c(pl.value, p2.value, ..., pk.value) 


#computing unconditional probabilities 
probs.matriz.name[1,] <- init.prob.name 

for(n in 2:nsteps.name) 

probs.matriz.name[n,]<- probs.matriz.name[n-1,]%*,tm.name 


#plotting probabilities against steps by state 
matplot (probs.matriz.name, type="1", lty=1, col=a:b, 
ylim=c (lower.value, upper.value) , ylab="probability", xlab="step") 


In addition, a legend should be added to match the lines with states’ numbers. 


legend(legend.position, c("state 1", "state 2",..., "state k"), 
lty=1, col=a:b) 


e The choices for legend. positionon the graph are: "topright", "right", 
"bottomright", "top", "center", "bottom", "topleft", "left", and 
"bottomleft". 


SIMULATION 1.1. Consider the Markov chain discussed in Examples 1.1-1.4, 
1.6, and 1.8. Recall that the transition probability matrix for this chain is 
specified as 

0.7 O01 0.2 

0.0 0.6 0.4 

0.5 0.2 0.3 
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Below we simulate two trajectories of this chain with the initial state cho- 
sen at random, that is, with the probability vector (1/3, 1/3,1/3). First, we 
demonstrate how to utilize the built-in function rmarkovchain(). 


#specifying transition probability matrix 
tm<- matrix(c(0.7, 0.1, 0.2, 0.0, 0.6, 0.4, 0.5, 0.2, 0.3), 
nrow=3, ncol=3, byrow=TRUE) 


#creating Markov chain object 
library (markovchain) 
mc<- new("markovchain", transitionMatrix=tm, states=c("1", 
hg) 3) 
2 


#specifying total number of steps 
nsteps<- 25 


#specifying initial probability 
po<- c(1/3, 1/3, 1/3) 


#specifying matrix containing states 
MC.states<- matrix(NA, nrow=nsteps, ncol=2) 


#specifying seed 
set .seed (2443927) 


#simulating trajectories 

for (i in 1:2) 

stateO<- sample(1:3, 1, prob=p0) 

MC.states[,i]<- rmarkovchain(n=nsteps-1, object=mc, 
tO=stateO, include.t0=TRUE) 


#plotting simulated trajectories 

matplot(MC.states, type="1", lty=1, lwd=2, col=3:4, 
xaxt="n", yaxt="n", ylim=c(1,3), xlab="Step", ylab="State", 
panel.first=grid()) 


axis(side=1, at=c(1,5,10,15,20,25)) 
axis(side=2, at=1:3) 


points(1:nsteps, MC.states[,1], pch=16, col=3) 
points(1:nsteps, MC.states[,2], pch=16, col=4) 
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Now we present the code with a user-defined function that simulates trajecto- 
ries, bypassing the built-in rmarkovchain() function. We use the same seed 
as above and obtain the same trajectories. The code and graph follow. 


#creating user-defined function 
MC<- function(tm, pO, nsteps) { 
states<- numeric (nsteps) 

states[1]<- sample(1,1,prob=p0) 


for(t in 2:nsteps) { 
p<- tm[states[t-1],] 
states[t]<- sample(1,1,prob=p) 


return(states) 


#specifying seed 
set .seed (2443927) 


#simulating trajectories 

MC.states2<- matrix(NA, nrow=nsteps, ncol=2) 
for (j in 1:2) 

MC.states2[,j]<- MC(tm, pO, nsteps) 
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#plotting simulated trajectories 

matplot(MC.states2, type="1", lty=1, lwd=2, col=3:4, 
xaxt="n", yaxt="n", ylim=c(1,3), xlab="Step", ylab="State", 
panel.first= grid()) 


axis(side=1, at=c(1,5,10,15,20,25)) 
axis(side=2, at=c(1,2,3)) 


points(1:nsteps, MC.states2[,1], pch=16, col=3) 
points(1:nsteps, MC.states2[,2], pch=16, col=4) 


3-4 


State 
ine) 
1 


Step 


Now we compute iteratively probability vectors and plot them against the 
step number. The code and plot are given below. 


#specifying matrix containing probabilities 
probs<- matrix(NA, nrow=nsteps, ncol=3) 


#specifying total number of steps 
nsteps<- 15 
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#computing probabilities p_n 
probs[1,] <- pO 

for(m in 2:nsteps) 

probs [n,]<- probs[n-1,]%*%tm 


#plotting probabilities against steps by state 
matplot(probs, type="1", lty=1, lwd=2, col=2:4, 
ylim=c(0.2,0.5), xlab="Step ", ylab="Probability", 
panel.first=grid()) 

legend("right", c("State 1", "State 2", "State 3"), lty=1, 
lwd=2, col=2:4) 


0.50 + 
0.40 5 
A 
3 
oO 4 
2 
2 
ao 
0.30 
0.20 i i 
T T T T T T T 
2 4 6 8 10 12 14 
Step 


We also output the values of the probabilities to see that they converge to the 
steady state (0.4651, 0.2558, 0.2791) on step 14. 


> probs 
[,1] [,2] [,3] 
<lines omitted> 
[13,] 0.4650353 0.2558698 0.2790949 
[14,] 0.4650721 0.2558444 0.2790835 
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SIMULATION 1.2. We return to the Markov chain studied in Examples 1.5, 
1.7, and 1.9. First, we use the already-created object mc to generate three 
trajectories that start at a randomly chosen state. In the code that follows, 
we apply the rmarkovchain() function to simulate the trajectories. 


#specifying total number of steps 
nsteps<- 20 


#specifying initial probability 
pO0<- c(1/6, 1/6, 1/6, 1/6, 1/6, 1/6) 


#specifying matrix containing states 
MC.states<- matrix(NA, nrow=nsteps, ncol=3) 


#specifying seed 
set .seed (765881) 


#simulating trajectories 
for (i in 1:3) { 
stateO<- sample(1:6, 1, prob=p0) 
MC.states[,i]<- rmarkovchain(n=nsteps-1, object=mc, 
t0O=state0, 
include.t0=TRUE) 
} 


#plotting simulated trajectories 
matplot(MC.states, type="1", lty=1, lwd=2, col=2:4, xaxt="n", 
ylim=c(1,6), xlab="Step", ylab="State", panel.first=grid() ) 


axis(side=1, at=c(1,5,10,15,20)) 
points(1:nsteps, MC.states[,1], pch=16, col=2) 


points(1:nsteps, MC.states[,2], pch=16, col=3) 
points(1:nsteps, MC.states[,3], pch=16, col=4) 
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Next, we compute and plot the unconditional probabilities p, against n. As 
n increases, these probabilities converge to an invariant probability measure, 
which heavily depends on the initial state of the Markov chain. These limiting 
values will be the same for states 1 and 2 since they belong to the same class, 
for states 5 and 6 for the same reason, and separately for state 3, and for state 
4. From the theoretical viewpoint, the invariant measure for states 1 and 2 is 
(0.5882, 0.4118, 0,0,0,0). For states 5 and 6, it is (0,0,0,0, 0.4375, 0.5625). 
From state 3 the chain is equally likely to enter either {1,2} or {5,6}, 
therefore, the invariant measure is found as (0.5)(0.5882, 0.4118, 0,0,0,0) + 
(0.5)(0, 0, 0, 0, 0.4375, 0.5625) = (0.2941, 0.2059, 0, 0, 0.21875, 0.28125). From 
state 4, the chain will enter the class {5,6} directly with probability 0.4, or 
through state 3, with probability (0.6)(0.5) = 0.3, hence, it enters class {5,6} 
with the total probability 0.4+ 0.3 = 0.7, and enters class {1,2} only through 
state 3 with the probability (0.6)(0.5) = 0.3. As a result, the invariant measure 
for state 4 is (0.3)(0.5882, 0.4118, 0,0, 0,0) + (0.7)(0, 0, 0, 0, 0.4375, 0.5625) 

= (0.17646, 0.12354, 0, 0, 0.30625, 0.39375). 


We run the code below six times, every time choosing a different initial state. 
As the output, we present the graphs for each initial state, and print the vec- 
tor of probabilities for steps 28 through 30. 


#specifying total number of steps 
nsteps<- 30 
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#specifying initial state distribution (state 1) 
p0<- c(1,0,0,0,0,0) 

#(state 2) p0<- c(0,1,0,0,0,0), (state 3) p0<- 
c(0,0,1,0,0,0), 

#(state 4) p0<- c(0,0,0,1,0,0), (state 5) p0<- 
ciCOMOMO MORENO) is 

#(state 6) p0<- c(0,0,0,0,0,1) 


#specifying matrix containing probabilities 
probs<- matrix(NA, nrow=nsteps, ncol=6) 


#computing probabilities p_n 
probs[1,] <- pO 


for(m in 2:nsteps) 
probs[n,]<- probs[n-1,]%*/tm 


#plotting probabilities vs. step by state 
matplot(probs, main="Initial State 1", type="1", lty=1, 
lwd=2, col=1:6, ylim=c(-0.05, 1.1), xlab="Step", 
ylab="Probability", panel.first = grid()) 


legend("topright", c("State 1", "State 2", "State 3", "State 
4", "State 5", "State 6"), lty=1, lwd=2, col=1:6) 


Initial State 1 
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[,41] [,2] [,3] [,4] [,5] [,6] 

<lines omitted> 
[28,] 0.5882082 0.4117918 0) 0) 0) 0) 
[29,] 0.5882542 0.4117458 0) 0) 0) 
[30,] 0.5882220 0.4117780 0) 0) 0) 0) 


fo) 


Initial State 2 


State 1 
oe ame State 2 
State 3 
State 4 
State 5 
State 6 


> | 
= 0.6 4'-- - - To 
a 1 1 1 1 
§ 
Q i i i i 
Oo 0.4 4-- YO- eTHE Se See CIS So eee Te. 
0.24--f Lf eae popes: ete eee aauae ‘ 
oof HE 

0 5 10 15 20 25 30 

Step 
[,1] [,2] [,3] (,4] [,5] [,6] 


<lines omitted> 
[28,] 0.5882739 0.4117261 0) 0) 0) 0) 
[29,] 0.5882082 0.4117918 0) 0) 0) 
[30,] 0.5882542 0.4117458 0) 0) 


jo) 
oe) 
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Initial State 3 


State 1 
State 2 


1.0 5 


o © + A 
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Step 


[,6] 


[,5] 


C.2) [43] (,4] 


<lines omitted> 


0.2058630 


0.2058959 
0.2058729 


[, 1] 


0.2812504 
0.2812498 
0.2812501 


0.2187496 
0.2187502 
0.2187499 


0 
0 
0) 


0 
0 
0 


[28,] 0.2941370 
[29,] 0.2941041 
[30,] 0.2941271 
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[30,] 
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27 
[,1] [,2] [,3] [,4] [,5] [,6] 
<lines omitted> 

[28,] 0.1764540 0.1235460 0) 0) 0.3062501 0.3937499 
[29,] 0.1764822 0.1235178 0) 0) 0.3062500 0.3937500 
. 1764625 0.1235375 0) 0) 0.3062500 0.3937500 

Initial State 5 

State 1 
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[,5] 


<lines omitted> 


0 0 0 
0 0 0 
0 0 0 


0.4374994 
0.4375003 
0.4374998 


[,6] 


0.5625006 
0.5624997 
0.5625002 
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Initial State 6 
State 1 
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[,1] [,2] (,3] [,4] [,5] [,6] 
<lines omitted> 

[28,] 0) 0) 0) 0) 0.4374994 0.5625006 

[29,] 0 0 0) 0) 0.4375003 0.5624997 

[30,] 0) 0) 0) 0) 0.4374998 0.5625002 


a 


1.8 Applications of Markov Chain 


APPLICATION 1.1. In 1918, A. A. Markov (1856-1922), a Russian mathemati- 
cian after whom Markov chains are named, published an article where he 
analyzed sequences of vowels and consonants among the first 20,000 letters 
of “Eugene Onegin” by A. S. Pushkin.” Two silent letters in the Russian lan- 
guage that are indicators of the softness of a preceding sound and are neither 
vowels nor consonants were ignored. He cleverly argued that the appearance 
of vowels and consonants are remarkably dependent in such a way that this 
literary work may be approximated by what we now call a Markov chain. He 
went ahead and computed the one-step transition probability matrix for this 
chain with the state space S = {vowel (v), consonant (c)}. He calculated that 


?Markov, A. A. (1913). “An example of statistical investigation of the text Eugene Onegin 
concerning the connection of samples in chains.” (In Russian). Bulletin of the Imperial 
Academy of Sciences of St. Petersburg, 7(3): 153 — 162. 
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there are 8, 638 vowels in the text (respectively, 11,362 consonants) and 1, 104 
vowel-vowel sequences. It is also important to notice that the text starts with 
a consonant and ends with a vowel, meaning that nothing transitions into the 
first consonant and the total number of transitions into vowels adds up to the 
total number of vowels. Thus, there must be 8, 638—1, 104 = 7, 534 consonant- 
vowel sequences, 11, 362 — 7,534 = 3, 828 consonant-consonant sequences, and 
11,361—3, 828 = 7, 533 vowel-consonant sequences. The transition probability 
matrix can then be computed as 


Vv Cc 
1104 7533 
—<—<—<— 12 =—_—_—_ * 22 
pe v | goa? "8 G37 — O87 
7534 3828 
——_—_—_—_—_- — Pe i ————— . 
[ga tiBaa 


Now, out of curiosity, we compute the limiting probabilities, which, theoret- 
ically speaking, represent proportions of vowels and consonants in the text. 
8638 


Since we know the exact numbers, we compute ty = 39599 = 0.4319, and 


Te = 1 — 0.4319 = 0.5681. Now, resorting to R, we obtain the same values: 


#specifying the transition probability matrix 

tm<- matrix(c(0.1278, 0.8722, 0.6631, 0.3369), nrow=2, 
ncol=2, 

byrow=TRUE) 


#creating Markov chain object 

library (markovchain) 

mc<- new("markovchain", transitionMatrix=tm, states=c("v", 
i) 

#computing limiting probabilities 

steadyStates (mc) 


Vv Cc 
0.4319026 0.5680974 


To verify how accurate A. A. Markov was with his estimation of the one-step 
transition probability matrix, we ran the same analysis on the entire novel. 
First, we precleaned the text, removing everything besides the Cyrillic letters 
of the pre-1918 reform Russian alphabet. Then we ran the code presented 
below that changes the capitalization to lowercase, then removes blanks, line 
breaks, punctuation marks, and the soft and hard signs of the Russian alpha- 
bet, leaving all the cleaned analysis-ready string containing a total of 106, 508 
characters. To compute the combinations vv, cv, vc, and cc, we shift the 
string to the left by one position, by inserting a blank in the front and deleting 
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the last letter to preserve the length of the string. When we line up the lagged 
string 2, with the original string x2, we obtain the preceding and following 
letters in the string. After that, we apply simple Boolean logic to calculate 
the number of vowels and consonants and the number of combinations of all 
four types. The code and output follow. 


text <- read_file("./Onegin.txt") 


#text cleaning 

#gsub() = global substitution function=replaces all instances 
lowercase<- tolower (text) 

no.blanks<- gsub("","", lowercase) 

no.line.breaks<- gsub("\r\n", "", no.blanks) 

no.punctuation<- gsub("[[:punct:]]","",no.line. breaks) 
no.soft.signs<- gsub("5", "", no.punctuation) 

#removing all hard signs 


clean.string<- gsub("3","" 


, no.soft.signs) 


#splitting single string into characters 
x2<- strsplit(clean.string, "") 


#shifting string by one position 

no.last<- substr(clean.string, 1, nchar(clean.string) -1) 
first.blank<- str_c("", no.last) 

x1<- strsplit(first.blank,"") 


#Note: In pre-1918 Russian language "“%" was considered a 
vowel 


vowels<-c("a", Weill Weile. Worl. Wale a, Moy fey Men We. 
Noite Wig les Wen 

consonants<- C(O" ,, Wes te eas Wares Us aaa Wey Milt Warts Ui i 
ize Wirndts Mo Wow Westie May MS Me Weil ie ile Tres) 


#computing number of vowels, consonants, and four 
combinations 

for (counter in 1:nchar(x2)){ 

v<- ifelse(x2[[counter]] %in% vowels,1,0) 

c<- ifelse(x2[[counter]] %in% consonants,1,0) 
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vv<- ifelse(xi[[counter]] %in% vowels & x2[[counter]] %in% 
vowels,1,0) 

ve<- ifelse(xi[[counter]] %in% vowels & x2[[counter]] %in% 
consonants,1,0) 

cv<- ifelse(x1[[counter]] 4in{% consonants & x2[[counter]] 
Zin’, vowels,1,0) 

cc<- ifelse(x1[[counter]] 4in{ consonants & x2[[counter]] 
%iny consonants,1,0) 

} 


sum(v) 
46475 
sum(c) 
60033 
sum(vv) 
6368 
sum(vc) 
40107 
sum(cv) 
40107 
sum(cc) 
19925 


Note that all these numbers add up perfectly. The total number of vowels 
and consonants is 46,475 + 60,033 = 106,508. Since the text starts and 
ends with consonants, all vowels have leading and trailing letters and there- 
fore, we must have sum(v)=sum(vv)+sum(vc)=sum(vv)+sum(cv). Indeed, 
sum(vc)=sum(cv)=40,107 and sum(vv)+sum(vc) = 6,368+ 40, 107 = 46, 475 
=sum(v). Also, all consonants but the first one have leading letters and 
all consonants but the last one have trailing letters. Hence, we must have 
sum(c)-1=sum(cv)+sum(cc)=sum(vc)+sum(cc), which indeed holds since 


40, 107 + 19,925 = 60, 032 = 60,033 — 1. 


Turning these numbers into the one-step transition probability matrix, we ob- 
tain 


Vv Cc 
6368 40107 
40107 19925 
—_—_—_—_—_- — i 1 ———— = * 1 
© laogaa oO aogaa 
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Note that A. A. Markov used about one-fifth of the text and obtained pretty 
accurate estimates. 


APPLICATION 1.2. It is important to understand that it is virtually impossible 
to show that a given process is a Markov chain because we would need to show 
that the Markovian property (1.1) holds for all tuples: pairs, triples, quadru- 
ples, quintuples, etc. of successive observations. In his 1913 article (see the 
previous application), A. A. Markov went only as deep as looking at triples, 
and thus he could only conclude that the process is approximately Markovian 
(put in modern terms). 


On the other hand, showing that a given process is non-Markovian is much 
easier because it suffices to show that the definition (1.1) fails for some tuple. 
For instance, if we can show that for some fixed 21, 72, and is, P(X3 = i3|X, = 
iy, Xq = ig) F P(X3 = i3|X2 = ig), it would be enough to prove that the 
process is not a Markov chain. Below we give an example of such a process. 


Staying on the topic of literature, recall that in the famous 1980 movie “Shin- 
ing,” the main character Jack Torrance (played by Jack Nicholson) repeatedly 
typed on a typewriter the sentence “All work and no play makes Jack a dull 
boy.” Suppose he typed it up & times, where & is a very large number. If 
we turn this “composition” into a sequence of vowels (v) (a, e, i, 0, u), and 
consonants (c) (the remaining 21 letters), we will get a truly deterministic 
sequence with a repeating pattern: 


allworkandnoplaymakesjackadullboy|allworkandnoplay... 
veccvccveccvccvccvevccvccvevcccvc|veccvccvcccvccvc... 


We want to show that the Markovian property doesn’t hold, for instance, for 
the triples {ccc} and thus this chain is non-Markov. In the entire “composi- 
tion” there are 33k letters, of which 11k are vowels and 22k are consonants. 
Also, pattern {ccc} occurs 3k times, {ccv} occurs 8k times, {vcc} occurs 8k 
times, and {vcv} occurs 2k + k — 1 = 3k — 1 times because two such triples 
lie inside each sentence and one appears at each seam. Thus, we find 


P(X Cc, Xo Cc, X3 c) 
P(X =C, Xo = c) 


P(X c| X2 c, X1 c) 


P(X Cc, Xo Cc, X3 c) 
P(X Cc, Xo Cc, X3 v) + P(X, Cc, Xo Cc, X3 c) 


nan 


P(ccc) _ sk 3 
P(cev) +P(cce) 8k+3k 11 
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On the other hand, we estimate 


P(X2 = c) 
_ P(cec) o- P(vec _ 3k + 8k 
P(cec) +P ccv) + P(vec) + P(vev) 3k + 8k + 8k+3k—1 
11k 1 
= S594 Be for large k. 


Since 4 say. this chain doesn’t satisfy the definition (1.1), and hence we 


have shown that this process is not a Markov chain. 


APPLICATION 1.3. According to the Mendelian model of gene inheritance in 
humans, named after Gregor Johann Mendel (1822-1884), a specific genetic 
trait is determined by a pair of genes, that can be of three types: AA, Aa, or 
aa. During reproduction, an offspring inherits one gene of the pair from each 
parent, and genes are selected at random, independently of each other. 


Suppose gene a is a mutant gene or a person possessing this gene is a carrier 
of a disease. Consider the genotype of the offspring in successive generations if 
the second parent always has genotype AA. This can be presented as a Markov 
chain with the state space S = {AA, Aa, aa} and transition probability matrix 


AA Aa aa 
AA 1 0 O 

P= Aa |0.5 0.5 . 
aa 0 1 O 


Indeed, if parents have genes (AA, AA), then their offspring are bound to 
have genes AA with probability 1. If parents have genes (Aa, AA), they are 
equally likely to spit into AA or Aa, and finally, if parents have genes (aa, AA), 
their offspring with certainty will have genes Aa. The diagram for this model is 


#specifying the transition probability matrix 
tm<- matrix(c(1, 0, 0, 0.5, 0.5, 0, 0, 1, 0), nrow=3, ncol=3, 
byrow=TRUE) 


#transposing the transition probability matrix 
tm.tr<- t(tm) 
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#plotting the diagram for the Markov chain 

library (diagram) 

plotmat(tm.tr, pos=c(1,2), name=c("AA", "Aa", "aa"), 
arr.length=0.3, arr.width=0.1, box.col="light blue", 
box.lwd=1, box.prop=0.5, box.size=0.12, box.type="circle", 
cex.txt=0.8, lwd=1, self.cex=0.6, self.shiftx=0.17, 

self .shifty=-0.01) 


Note that states Aa and aa are transient states, and AA is the absorbing state. 
It means that in a long run, the gene a will disappear from the population. 
To convince ourselves, we compute the stationary distribution. 


#creating Markov chain object 

library (markovchain) 

mc<- new("markovchain", transitionMatrix=tm, 
states=c("AA", "Aa", "aa")) 


#computing stationary distribution 
steadyStates (mc) 


AA Aa aa 
1 0 O 


Let’s assume that initially the gene a is present in the 1% of the population in 
the combination aa, that is, p44 = 0.99, paa = 0, and pag = 0.01. We run the 
R code below to see how the population genetic composition changes from gen- 
eration to generation by computing recursively py+1 = pn - P,n = 1, 2,3, etc., 
with the initial condition p, = (0.99, 0, 0.01). 
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library (expm) 

geni<- c(0.99, 0, 0.01) 
gen<- gen1/*/tm 

for (n in 2:10) { 

print (n) 

print (round(gen, digits=10)) 
gen<- gen/*/,tm 


} 

n DAA PAa Paa 
1 0.99 0 0.01 
2 0.99 0.01 0 

3. 0.995 0.005 0 

4 0.9975 0.0025 0 

5 0.99875 0.00125 0 

6 0.999375 0.000625 0 

7  0.9996875 0.0003125 0 

8 0.9998438 0.00015625 0 

9 0.999921875 0.000078125 0 
10 0.9999609375 0.0000390625 0 


Note that already in the second generation the gene type aa disappears, and 
is transformed into the hybrid type Aa, and that after as many as ten gener- 
ations, the gene a is still lingering on in the population. 


APPLICATION 1.4. In this application, we present yet another chain that 
is not Markov. It has to do with weather conditions. On an intuitive level, 
weather tomorrow depends not just on today’s weather but on the weather 
for several past days, if not the entire history of weather conditions in the re- 
gion. To support this intuitive supposition numerically, we downloaded from 
kaggle.com an open-access historical hourly weather data for 2012-2017 (file 
“weather description.csv”), focused only on the column for Los Angeles, and 
clumped the weather conditions into the four categories 


S = {s="sky clear", c="cloudy", f="fog",r="rain"}. 


We then found empirically the conditional probability of clear sky tomorrow, 
given clear skies yesterday and today, 


nA 


P(sss) 
P(sss) + P(ssc) + P(ssf) + P(ssr) 
and the conditional probability of clear sky tomorrow, given clear sky today, 


= 0.9647, 


P(X3 s |X» s, X41 


na 


~ P(sss) + P(css) + P(fss) + P(rss) 


P(X3 =s|X2=s)=—x = = = 0.9356. 
P(sss) + P(css) +---+P(rsr) 
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Since these two estimates are not the same, we concluded that hourly weather 
conditions don’t form a Markov chain. The complete R code and output follow. 


weather.data<- read.csv("./weather_description.csv", 
header=TRUE, sep=",") 


LA<- weather.data$Los.Angeles 


X3<- ifelse(LA=="sky is clear", "clear", ifelse(LA %in% 
c("broken clouds", "few clouds", "overcast clouds", 
"scattered clouds"), "clouds", ifelse(LA %in% c("light 
intensity drizzle", "dust", "fog", "haze", "mist", "smoke", 
"drizzle"), "fog", "rain"))) 


library(Hmisc) #Harrell Miscellaneous packages 
X2<- Lag(X3,shift=1) 
X1<- Lag(X3,shift=2) 


sss<- ifelse(X1=="clear"& X2=="clear"& X3=="clear",1,0) 
css<- ifelse(X1=="clouds"& X2=="clear"& X3=="clear",1,0) 
fss<- ifelse(X1=="fog"& X2=="clear"& X3=="clear",1,0) 
rss<- ifelse(X1=="rain"& X2=="clear"& X3=="clear",1,0) 
ssc<- ifelse(X1=="clear"& X2=="clear"& X3=="cloudy",1,0) 
csc<- ifelse(X1=="clouds"& X2=="clear"& X3=="clouds",1,0) 
fsc<- ifelse(X1=="fog"& X2=="clear"& X3=="clouds",1,0) 
rsc<- ifelse(X1=="rain"& X2=="clear"& X3=="clouds",1,0) 
ssf<- ifelse(X1=="clear"& X2=="clear"& X3=="fog",1,0) 
csf<- ifelse(X1=="clouds"& X2=="clear"& X3=="fog",1,0) 
fsf<- ifelse(X1=="fog"& X2=="clear"& X3=="fog",1,0) 
rsf<- ifelse(X1=="rain"& X2=="clear"& X3=="fog",1,0) 
ssr<- ifelse(X1=="clear"& X2=="clear"& X3=="rain",1,0) 
csr<- ifelse(X1=="clouds"& X2=="clear"& X3=="rain",1,0) 
fsr<- ifelse(X1=="fog"& X2=="clear"& X3=="rain",1,0) 
rsr<- ifelse(X1=="rain"& X2=="clear"& X3=="rain",1,0) 


#computing P(X3=s|X2=s,X1=s) 
sum(sss)/sum(sss+ssctssf+ssr) 
0.9647471 


#computing P(X3=s|X2=s) 
sum(ssst+csst+tfss+rss) /sum(sss+csstfsstrsstssctcsctfsctrsctssf 
t+csft+fsftrsftssr+csrt+fsrtrsr) 


0.9355981 
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TT 


Exercises 


EXERCISE 1.1. A Markov chain has a one-step transition probability matrix 


1 2 3 
1 70.3 0.4 0.3 
2 [02 0.3 05) 
3 10.8 0.1 0.1 


Compute the following probabilities: 

(a) P(X3 2| Xo 1, X\ 2, Xo 3). 

b) P(X4 = 3|Xo = 2, X3 = 1). 

c) P(Xo = 1, X1 = 2, Xo = 3, X3 = 1). Assume P(Xo = 1) = 
d) P(Xo = 1, X1 = 2, X3 = 3, X5 = 1). Assume P(Xp = 1) = 1. 


EXERCISE 1.2. Consider a Markov chain with the transition probability matrix 


1 2 3 4 = «5 
1.0 0.0 0.00 0.0 0.0 
0.5 0.0 0.0 0.0 0.5 
0.2 0.0 0.0 0.0 0.8]. 
0.0 0.0 1.0 0.0 0.0 
0.0 0.0 0.0 1.0 0.0 


orRWNH 


(a) Plot a diagram of the Markov chain. 

(b) Identify all transient and recurrent classes. Identify all absorbing and re- 
flective states. Find the period of each state. 

(c) Simulate three trajectories of the chain that start at a randomly chosen 
state. Comment on what you see. 

(d) Find the steady-state probabilities and interpret them. Is it an ergodic 
chain? 

(e) Plot the unconditional probabilities at time n against the time and com- 
ment on how fast the probabilities converge to the steady-state distribution. 


EXERCISE 1.3. Consider a Markov chain with the one-step transition proba- 
bility matrix 
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1 23 4 5 6 7 
1 010 0 0 0 0 
2 100 0 0 0 0 
3] 0 0 0 04 0.2 0.2 0.2 
4/ 0 00 O 0.2 0.4 0.4 
5/03 0 0 01 03 O01 0.2 
6] 0 0 0 0.2 0.2 0.3 0.3 
7L0 0 0 05 0.2 0.2 0.1 


(a) Plot a diagram of the Markov chain. 

(b) Identify all recurrent and transient classes. Find their periods. Are there 
any absorbing and reflecting states? 

(c) Simulate two trajectories of the chain that start at a randomly selected 
state. Discuss what you see in the plot. 

(d) Calculate the limiting probabilities and interpret them. Is the chain er- 
godic? 

(e) Plot the unconditional probability vectors py, against n and comment on 
the speed of convergence to the limiting distribution. 


EXERCISE 1.4. Consider a Markov chain with the one-step transition proba- 
bility matrix 


1 2 3. °¢C«4 5 


170.1 02 03 O 04 
2] 0 05 05 0 0 
3 | 0 1 OO O- O}- 
4/0 0 0 0 1 
5L0 0 O 06 0. 


(a) Plot the diagram of the Markov chain. 

(b) Find all recurrent and transient classes and their periods. Are there any 
absorbing or reflecting states? 

(c) Simulate several trajectories of the Markov chain and discuss the patterns 
that you see. 

(d) Show that the chain is non-ergodic because there are two invariant prob- 
ability measures. Which one of them is the stationary distribution? 

(e) Plot the graphs of unconditional probabilities against time, assuming suc- 
cessively that the chain starts in states 1, 2, 3, 4, and 5. Interpret each graph. 


EXERCISE 1.5. In a box there are two red (R), four blue (B), and eight green 
(G) balls. One ball is drawn at a time and its color is noted. Consider the 
stochastic process {X,,, n = 1,2,...} with the state space S = {R, B,G}. 
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(a) Show that this process is not a Markov chain, if the drawing is done with- 
out replacement. Hint: Show, for instance, that P(X3 = G|X, = R,X2 = 
B) #P(X3 =G|X1 =G, X2 = B). 

(b) Show that this process is a Markov chain, if the drawing is done with 
replacement. 


EXERCISE 1.6. Consider a sequence of heads and tails obtained by a series 
of independent flips of a fair coin. Show that it can be modeled by a Markov 
chain with the state space S = {H,T}. Find the transition probability matrix 
and the limiting distribution. 


EXERCISE 1.7. Assume that the usage of vowels and consonants in “Moby 
Dick” by Herman Melville can be modeled by a Markov chain. 

(a) Find the transition probability matrix for Chapter 1 of this novel. Cal- 
culate the limiting probabilities and verify that they are equal to the overall 
proportions of vowels and consonants in the text. 

(b) Do states in Chapter 2 follow the same transition probability matrix as 
those in Chapter 1? 


EXERCISE 1.8. A student at a secretarial school typed the sentence “The quick 
brown fox jumped over the lazy dog” 500 times. Show that the resulting text 
cannot be modeled as a Markov chain. 


EXERCISE 1.9. Consider the Mendelian gene inheritance model introduced in 
Application 1.3. Suppose that the second parent is chosen randomly from the 
gene pool with all types AA, Aa, or aa. 

(a) Show that the genotype of the offspring follows a Markov chain with the 
state space S = {(AA, AA), (AA, Aa), (AA, aa), (Aa, AA), (Aa, Aa), (Aa, aa), 
(aa, AA), (aa, Aa), (aa,aa)} and the transition probability matrix 


(AA, AA) (AA, Aa) (AA,aa) (Aa, AA) (Aa, Aa) (Aa,aa) (aa, AA) (aa, Aa) (aa, aa) 
1 0. 


(AA, AA) /3 1/3 1/3 0 0 0 0 0 


(AA, Aa) 1/6 1/6 1/6 1/6 1/6 1/6 0 0 | 
(AA, aa) 0 0 0 1/3 1/3 1/3 0 0 0 
(Aa, AA) 1/6 1/6 1/6 1/6 1/6 1/6 0 0 0 
(Aa, Aa) 1/12 1/12 1/12 1/6 1/6 1/6 1/12 1/i2s1/12 
(Aa, aa) 0 0 0 1/6 1/6 1/6 1/6 1/6 1/6| 
(aa, AA) () 0 () 1/3 1/3 1/3 0 0 0 
(aa, Aa) 0 0 0 1/6 1/6 1/6 1/6 1/6 1/6 
(aa, aa) 0 0 0 0 0 0 1/3 1/3 1/3 


(b) Determine the transient and recurrent classes of the Markov chain. 
(c) Find the stationary distribution. What is the steady-state genetic com- 
position for both parents? 
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(d) Which initial state achieves the stationary distribution in the smallest 
number of generations? Which in the largest’? Assume the precision of four 
correct decimals after rounding. 


EXERCISE 1.10. Refer to Application 1.4. Consider the data in the file 
“weather description.csv.” Choose a city other than Los Angeles and con- 
duct the analysis similar to the one given the application. 


EXERCISE 1.11. On an intuitive level, pollution level for a region doesn’t fol- 
low a Markov chain as the pollution level tomorrow depends on pre-history 
and not just on today’s level. But suppose that for some areas, air quality sta- 
tus (good/unhealthy /hazardous) depends only on those in the previous two 
days, and not in earlier days. Show that in this case, we can look at two days 
at a time and model data as a Markov chain. 


EXERCISE 1.12. Consider a simplified monopoly game with only five squares 
and respective incomes of $200, $0, -$75, $105, and -$130. A player starts at 
the first square, rolls a fair die once, and moves forward as many steps as the 
die shows. 

(a) Argue that this game can be modeled as a Markov chain and find its 
transition probability matrix. 

(b) Compute the steady-state probability of each square, and find the long- 
run winning of the player. 


EXERCISE 1.13. Suppose that road traffic conditions can be modeled as a 
Markov chain with the state space S = {light, heavy, jammed}, and suppose 
that traffic conditions change every 20 minutes. Assume that between 1PM 


0.4 0.4 0.2 
and 4PM, the transition probability matrix is |0.3 0.5 0.2}, whereas be- 
0 056 0.5 
0.1 0.5 0.4 
tween 4PM and 6PM it changes to |0.1 0.3 0.6]. 
0 O11 0.9 


(a) If the traffic starts with the light state at 1PM, what is the distribution 
of the states at 6PM? 
(b) Simulate 10,000 trajectories to verify the result of part (a). 


EXERCISE 1.14. A certain species of shrubs has four states: state 1 if it is 
sustainable, state 2 if it is threatened, state 3 if it is endangered, and state 
4 if it is extinct. Plant assessment surveys are done at regular time intervals. 
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Transitions between states are modeled by a Markov chain with the transition 
probability matrix 


1 2 3 +4 
1 bee 0.2 0.1 aa 

0.7 0.2 0.1 0.0 
3 Od 03 04-02)" 
4 10.0 0.0 0.0 1.0 


(a) Assuming that a shrub is initially sustainable, simulate several trajecto- 
ries of the Markov chain. 

(b) Find the probability that initially sustainable shrub will eventually be- 
come extinct. 


EXERCISE 1.15. A music instrument store is open every day of the week except 
Monday. During that day, if the inventory count is below 3, more instruments 
are ordered, so that by Tuesday morning there are 7 instruments in stock. If 
3 or more instruments are in stock, then no action is taken. The number of 
instruments sold during the business days is a Poisson random variable with 
a mean of 4. Any demand that cannot be satisfied is lost. (a) Argue that the 
inventory each Tuesday morning can be modeled as a Markov chain. Find its 
state space and the one-step transition probability matrix. 

(b) Generate inventory trajectories, assuming that the initial inventory size 
is randomly chosen. 

(c) Suppose one week there are 7 instruments in stock on Tuesday morning. 
Compute the probability that there will be 7 instruments in stock also on each 
of the three subsequent Tuesday mornings. 

(d) The weekly storage cost is $5 per instrument that is in the store on Tues- 
day morning. Compute the long-run expected weekly storage cost. 
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2 
Random Walk 


2.1 Definition of Random Walk 


Consider a random process which state space comprises all integers on a real 
line. The process starts at zero and transitions either one step to the right 
with probability p or one step to the left with probability 1 — p. This pro- 
cess is called a simple random walk. It is called symmetric if p = 1/2 and 
asymmetric, otherwise. It is also termed infinite because it is defined on an 
infinite set of integers. If a random walk occurs on a finite subset of integers, 
it is termed a finite-state random walk or a random walk on a finite grid or a 
bordered random walk. 

In this chapter, we also consider some variations of a random walk, such 
as two-, three-, and higher-dimensional random walks, and random walks on 
graphs. Note that in general, random walks don’t have to start at the origin. 
They can start at any randomly chosen starting point. Below we give formal 
definitions of all these processes. 

A simple (asymmetric, one-dimensional, infinite) random walk! is a 
special case of a Markov chain which state space consists of integers 
S = {0,+1,+2,...}, and the transition probabilities are of the form pi i41 = 
P(Xn41 =iti+ 1|Xn = i) = p and Pii-1 = P(Xn41 =4- 1|Xn = i) = 
1l—p, 1=0,+1,+2,.... The transition probability matrix for a random walk 
is 


SB ovyvonon.: 
o's oo 2 


A symmetric random walk in two dimensions is defined as a random walk on 
an integer lattice that moves right or left or up or down with a probability 


1The term was first used in Pearson K. (1905). “The Problem of the Random Walk”. 
Nature, 72: 294. 
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1/4. Likewise, a symmetric random walk in d dimensions is a random walk 
on the d-dimensional integer lattice, with equiprobable moves in 2d directions 
(with probability 1/(2d)). 


Some variations of a simple random walk include a random walk with loops 
(or delays), in the sense that the allowed moves are to the right with proba- 
bility p, to the left with probability qg, and remaining in the same state with 
probability 1 — p — q. Also, in two dimensions, it might be allowed to move 
diagonally as well as to stay in the same place, so each move has a probabil- 
ity of 1/9. Or a finite-state random walk might be defined on a finite integer 
grid with the boundary (or border, or barrier) states being either reflecting 
or absorbing. In addition, it is possible to define a random walk!on a graph, 
where at every state, the process chooses among all neighboring states with 
equal probability. Below we consider some examples. 


EXAMPLE 2.1. A gambler either wins $5 with probability 0.55 or loses $5 
with probability 0.45. He starts playing with $50 and plays until he either 
goes broke or doubles his original amount. This is an example of a finite one- 
dimensional random walk with absorbing barriers, since, once entered, the 
states $0 and $100 are never left. The transition probability matrix is 


$0 $5 $10 ... $45 $50 $55 $95 $100 
$0 1 0 0... 0 0 0 0 0 
$5 | 0.45 0 0.55 ... 0 0 0 0 0 
$10 0 0.45 0 0 0 0 0 0 
$45 0 0 08 Ga 0 0.55 Ovsae 0 0 
$50 0 0 0 ... 0.45 0 O55 ... 0 oO} 
$55 0 0 Oe 24 0 0.45 On hac 0 0 
$95 0 0 OF net 0 0 0... 0 0.55 
$100 0 0 0... 0 0 07. 0 1 


If, for example, a gambler who goes broke can take a credit that brings him 
back to the $50 fortune, the state $0 is no longer an absorbing state, but 
is, in fact, a reflecting state. On the other end of his wealth spectrum, once 
the gambler reaches the $100 fortune, he returns the credit (with, say, 10% in- 
terest) and is bounced to $45, and thus the $100 state is reflecting as well. 


EXAMPLE 2.2. Suppose that a mouse is running through a maze with rooms 
A, B,C, D, and E, and two exits (see the illustration by Shayan Khatri). The 
mouse runs along the passages, and in any given room, it decides randomly 
which passage to take. 
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This motion can be modeled as a random walk on a graph. Exit is an absorb- 
ing state. State A has four neighboring states: Exit, B, C, and FE, and thus 
the mouse can go to either one of them with a probability of 1/4. State B has 
two neighboring states, A and C’, and the mouse goes to either one with a 
probability of 1/2. State C' has connections to all the other four states, giving 
the mouse a probability of 1/4 to choose the direction. State D connects to 
three states, C', FE, and Exit, so the mouse selects the state to go to next 
with a probability of 1/3. Finally, from state E the mouse can reach three 
states, A, C, or D, with probability 1/3 each. The one-step transition matrix 
summarizes these probabilities. 


Evit A B C OD 5B 
Exit i t £ & & 
A| 1/4 0 1/4 1/4 0 1/4 
B 0 1/2 0 1/2 0 6 
C 0 1/4 1/4 0 1/4 1/4) 
D| 1/3 O 0 1/3 O 1/3 
E Os 0 178-18 0 
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2.2. Must-Know Facts About Random Walk 


Denote by {X,, n = 0,1,2,...} an asymmetric one-dimensional random walk 
that moves in the positive direction with probability p. Assume that the ran- 
dom walk starts at zero, that is, Xp = 0. We formulate some interesting results 
and present them as propositions. 
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PROPOSITION 2.1. (MEAN AND VARIANCE OF A RANDOM WALK). The 


mean and variance of a random walk at time n are E(X,,) = (2p — 1)n and 
Var(X,) = 4p(1 — p)n. 


Proor: We can write X, = Z,+---+2Z, where Z;’s are independent random 
variables with the binary distribution: 


1, with probability p, ; 
Lee ; a t= Ay ag 
—1, with probability 1 — p, 


Note that E(Z;) = (1)(p) + (-1)(1 — p) = 2p — 1, and Var(Z,) = (1)?(p) + 
(—1)?(1 — p) — (2p — 1)? = 1—- (2p — 1)? = 4p — 4p? = 4p(1 — p). Thus, 
we compute E(X,) = E(Z,) + ---+E(Z,) = (2p — 1)n, and Var(X,) = 
Var(Z1) +---+ Var(Z,) = 4p(1— p)n. 


Note that for a symmetric random walk, p = 1/2, and hence, E(X,,) = 0 and 
Var(X,) =n, and after n steps, a typical distance from the origin is on the 
order of ./n. 


PROPOSITION 2.2. (PROBABILITY OF A RETURN). The probability that a 
one-dimensional random walk returns to the starting point in exactly n steps 


is ( ple _ p)r/? if n is even, and zero, otherwise. 
n 


PRooF: First of all, note that the random walk can return to the starting 
point only in an even number of steps, half of which it should be moving to 
the right (with probability p), and half, to the left (with probability 1 — p). 


There are ( » paths with this property, hence: 
n 
n 
P(X, =0|Xo = 0) = n/2 
0, if n is odd 


ora - p)r/?, if n is even, 


Note that the random walk can cross the starting point several times before 
ending there in exactly n steps. 


The proofs of the next two propositions are omitted as they lie beyond the 
scope of this book. 


PROPOSITION 2.3. (RECURRENCE VS. TRANSIENCE OF 1D RANDOM 
WALK). A one-dimensional random walk will come back to the origin in- 
finitely many times with probability one if and only if p = 1/2. This means that 
a symmetric one-dimensional random walk is recurrent, while an asymmetric 
one-dimensional random walk is transient. It will, with a positive probability, 
come back only finite number of times and will eventually wander away. 
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PROPOSITION 2.4. (RECURRENCE VS. TRANSIENCE OF d-DIM RANDOM 
WALK). A two-dimensional random walk is recurrent if and only if it is sym- 
metric (i.e., it goes up or down or left or right with probability 1/4). Any 
random walk (symmetric or not) in 3D or a higher dimension is transient. It 
will eventually wander away from the origin with a positive probability. 


Thence, a bug crawling randomly along a railroad track will visit every inch 
of it infinitely many times. Likewise, a King moving randomly on an infinite 
chessboard will visit every square infinitely many times, whereas, say, a drone 
performing a random walk in the sky will eventually go into outer space, never 
to be seen again. 
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2.3. Simulations in R 


SIMULATION 2.1. Below we simulate three trajectories of a one-dimensional 
random walk. We assume the walk starts at zero, and specify p as 0.6 and the 
number of steps as 25. 


#specifying parameters 
ntraj<- 3 

p<- 0.6 

nsteps<- 25 


#specifying seed 
set .seed (45568223) 


#defining walk as matrix 
walk<- matrix(NA, nrow=nsteps, ncol=ntraj) 


#simulating trajectories 

for (j in 1:ntraj) { 

walk[1,j]<- 0 

for (i in 2:nsteps) 

walk[i,j]<- ifelse(runif(1)<p, walk[i-1,j]+1, walk[i-1,j]-1) 
} 


#plotting trajectories 

matplot(walk, type="1", lty=1, lwd=2, col=2:4, 
ylim=c(range(walk)), xlab="Step", ylab="Position", 
panel. first=grid()) 
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points(1:nsteps, walk[,1], pch=16, col=2) 
points(1:nsteps, walk[,2], pch=16, col=3) 
points(1:nsteps, walk[,3], pch=16, col=4) 


Position 


SIMULATION 2.2. The code below simulates and plots a two-dimensional ran- 
dom walk with a total of 10,000 steps, emanating from the origin. 


#specifying number of steps 
nsteps<- 10000 


#specifying seed 
set .seed (607335) 


#defining walk as matrix 
walk<- matrix(NA, nrow=nsteps, ncol=2) 


#setting starting point 
walk[1,]<- c(0,0) 


#definiting random steps 
rstep<- matrix(c(1, 0, -1, 0, 0, 1, 0, -1), nrow=4, ncol=2, 
byrow=TRUE) 
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#simulating trajectories 
for (i in 2:nsteps) 
walk[i,]<- walk[i-1,] + rstep[sample(1:4, size=1) ,] 


#plotting trajectories 

plot (x=walk[,1], y=walk[,2], type="1", col="blue", 
xlim=range(walk[,1]), ylim=range(walk[,2]), xlab="x", 
ylab="y", panel. first=grid() ) 


#adding starting point 
points(cbind(walk[1,1], walk[1,2]), pch=16, col="green", 
cex=2) 


#adding ending point 
points (cbind(walk[nsteps,1],walk[msteps,2]), pch=16, 
col="red", cex=2) 
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SIMULATION 2.3. The following code simulates and plots a three-dimensional 
random walk with 5,000 steps. 


#specifying number of steps 
nsteps<- 5000 


#specifying seed 
set .seed (830126) 


#defining walk as matrix 
walk<- matrix(NA, nrow=nsteps, ncol=3) 


#setting starting point 
walk[1,]<- c(0,0,0) 


#defining random steps 
rstep<- matrix(c(1,0,0,-1,0,0,0,1,0,0,-1,0,0,0,1,0,0,-1), 
nrow=6, ncol=3, byrow=TRUE) 


#simulating trajectories 
for (i in 2:nsteps) 
walk[i,]<- walk[i-1,]+rstep[sample(1:6, size=1) ,] 


#plotting trajectories 
library (plot3D) 


lines3D(walk[,1], walk[,2], walk[,3], col="blue", 
xlim=range(walk[,1]), ylim=range(walk[,2]), 
zlim=range(walk[,3]), xlab="x", ylab="y", zlab="z", bty="b2", 
ticktype="detailed") 


#adding starting point 
points3D(x=walk[1,1], y=walk[1,2], z=walk[1,3], add=TRUE, 
pch=16, col="green", cex=2) 


#adding ending point 
points3D(walk[nsteps,1], walk[nsteps,2], walk[nsteps,3], 
add=TRUE, pch=16, col="red", cex=2) 
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2.4 Applications of Random Walk 


APPLICATION 2.1. (GAMBER’S RUIN PROBLEM). Gambling is perhaps the 
oldest area of application of random walks. And the most famous is the Gam- 
bler’s Ruin Problem. A version of the gambler’s ruin problem has been formu- 
lated as early as 1656, in correspondence between Blaise Pascal and Pierre de 
Fermat. 


Suppose a gambler starts with a fortune of $i and will move up $1 with prob- 
ability p or down $1 with probability g = 1 — p until he is either broke or 
reaches the fortune of $N. What is the probability that he goes broke? 


To answer this question, we will compute the complementary probability 
of reaching the fortune of $N. Denote by P,; the probability of winning 
$N if a gambler starts with $7. Conditioning on the outcome of the first 
move, we can write the recurrence relation: P; = gPj;-; + pPj+1 with the 
boundary conditions Pg = 0 and Py = 1. We can write this relation as 
pPj+qP; = qPj-1+pPj41 and rewrite as Pj41 —P; = q(P; _ Pea): From 
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here, we see that Pz — P; = 
2 

(4) Pagan PG — Py = (4 

P;-P,= [e+ (4)? 4-4 


Pp 
p, Tae Pi ifa/p Al, 
iP,, if g/p = 1. 


(Pi — Po) =/Pi, P3—-P2= 4(P2—P1) = 


“TP. Summing up the identities, we obtain 


aa P,. From here, 


° 


To find Pi, we use the boundary condition Py = 1. It yields 


1-q/ ‘ 
P, = TCT if q/p # 1, 
1/N, if g/p = 1. 
Consequently, if gambling is modeled as a symmetric random walk (with 
p =q=1/2), the probability of reaching the fortune $N is i/N, and thus, the 
probability of ruin is (N —7)/N. If the model is an asymmetric random walk, 
_ a 
the probability of reaching $N is P; = MeN, 
1 — (q/p) 
(a/p)' = (a/p)* 
1— (q/p)% 


Also, it can be shown (see Exercise 2.7) that the expected number of games 
that the gambler plays until he reaches $N or goes bankrupt is 


and hence, the probability 


of ruin is 1 — P; = 


—p  1—(q/p) Pp q-p ° 
i(N —i), if g/p = 1. 


Let us see how it plays out with some specific values. In Example 2.1, the gam- 
bler started with $50, goes up $5 with probability 0.55, or goes down $5 with 
probability 0.45. He would end up with either $100 or $0. Since the increment 
is $5, in our notation this translates into i = i. N = 20, and p = 0.55. The 
Sse = 0.8815 and the probability 
of ruin is, respectively, 1 — 0.8815 = 0.1185. 


4 = 


a a a if q/p # 1, 


probability of reaching $100 is 


10-—(2 881 
As for the expected number of games, we compute E; = eos) a 


76.3. That is, the gambler plays, on average, 76.3 games. 


We can verify these probabilities and the expected number of games empiri- 
cally, by running the following R code that simulates 100,000 trajectories and 
counts how many of them ended in 20, how many ended in 0, and keeps track 
of the total number of games played until the end is reached. These values are 
then averaged over the total number of trajectories. 
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#specifying parameters 


p<- 0.55 
i<x- 10 
N<- 20 


ntraj<- 100000 


#defining walk as vector 
walk<- c() 


#setting counters 
nNs<- 0 

nzeros<- 0 
ngames<- 0 


#setting seed number 
set .seed (30112443) 


#simulating trajectories until hitting N or 0 

for (j in 1:ntraj) { 

walk[1]<- i 

k<- 2 

repeat { 

walk[k]<- ifelse(runif(1)<p, walk[k-1]+1, walk[k-1]-1) 
ngames<- ngames + 1 


if (walk[k]==N) { 
nNs<- nNsti 
break 
} 


else if (walk[k]==0) { 
nzeros<- nzerosti 


break 
} 

k<- kt+1 

} 

} 


print (prob.Ns<- nNs/ntraj) 
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0.88279 


print (prob.zeros <- nzeros/ntraj) 
0.11721 

print (mean.ngames<- ngames/ntraj) 
76.18866 


Next, we can plot the graph of the probabilities as a function of p, for our 
specific values of 7 = 10 and N = 20. The syntax and the graph follow. The 
green curve depicts the probability of reaching N, whereas the red one dis- 
plays the probability of ruin. 


p<- seq(0.35,0.65,0.001) 


i= 10) 
N<- 20 
q<- 1-p 


p-ruin<- ifelse(p==0.5, (N-i)/N, ((q/p)i-(q/p)*N)/(1-(q/p) 
AN)) 


#ploting the graphs 
plot(p, p.ruin, type = "1", lwd=2, col = "red", xlab="p", 
ylab="Probability", panel.first = grid()) 


lines(p, 1-p.ruin, lwd=2, col = "green") 


legend("right", c("Probability of $0", "Probability of $N"), 
lty=1, col=2:3) 


1.0 +--; pcp == ~~ =H 'slalalaiaialaiatatets ‘alalaialalalaiaiaietalt jose a Te seo 
OB tacbicssccscscslaccossscccchs Meee cscsspecsseefosdicscecatetedisancseecsseqes 
= 
8 064--! ene eer eye Sater eee eee: Catan: a eee : eee tees We seco ed oe 
° ' Probability of $0 
S ‘ Probability of $N 
0.4 4--}----------- banananene ee pecenesofe- Ao -------5 ee ee 
God nctasienseeres Licance sesetasghtaoeed a oe ee eee = 
0.0 4-- a pases — ene — 
T Tt T Tt T T T 
0.35 0.40 0.45 0.50 0.55 0.60 0.65 
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Note from the graph that the probability p doesn’t have to be very small for 
the ruin to happen almost certainly. For p ranging between 0.4 and 0.6, the 
probability of ruin goes down from almost 1 to almost 0. Respectively, the 
probability of reaching N increases from almost 0 to almost 1. 


Finally, we plot the graph of the expected number of games against p. The 
code and the output are presented below. 


p<- seq(0.01,1,0.01) 


ie lle 
N<- 20 
q<- 1-p 


E.ngames<- 

ifelse(p==0.5,i*(N-i) , (i-N*(1-(q/p)i)/(1-(q/p) “N))/(1-2*p) ) 
plot(p, E.ngames, type="1", lwd=2, col="green", xlab="p", 
ylab="Expected number of games", panel.first=grid() ) 


100 + 


foe) 
oO 
1 


Expected number of games 
o 
[o) 


20 4-: 


Note that the maximum of this function is achieved at p = 1/2 and the max- 
imum value is i(N — i) = (10)(20 — 10) = 100. Also, the graph is symmetric 
due to the starting point being right in the middle, i.e., i = N/2. 
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APPLICATION 2.2. (RANDOM WALK ON A GRAPH). Consider the random 
walk through the maze introduced in Example 2.2. Suppose the mouse starts 
in room C and spends one second on each transition from room to room (or 
Exit). We want to calculate how many seconds, on average, the mouse spends 
in the maze before exiting. We argue as follows. The mouse will spend exactly 
k seconds in the maze if it exits in exactly k transitions. So, we need to find 
the probability to transition from room C' to an Exit in exactly k steps. Let 
P denote the one-step transition probability matrix. Then 


(0,0, 0, 1,0,0) P* (1,0,0,0,0,0)~? 


gives the probability to transition from vertex C to Exit in k or fewer steps, 
and thus, 


(0,0,0, 1,0, 0) (P' = Pet) (1,0,0,0,0,0)~? 


is the probability to transition from C to Exit in exactly k steps. Conse- 
quently, the formula for the expected time that the mouse spends in the maze 
before it reaches an Exit is 


E(time to exit) = (0,0,0,1,0,0) (YP + (2)(P2 — P) 


(3)(P3 — P?) 4 (4)(P4- P>) +...) (1,0,0,0,0,0)-2. 


We run an R code to estimate this expected value. Convergence to the six 
decimal places that R outputs is achieved with the first 172 terms. Adding 
more terms doesn’t change the output. 


#specifying transition probability matrix 

tm<- matrix(c(1,0,0,0,0,0,1/4,0,1/4,1/4,0,1/4,0,1/2,0, 
1/2,0,0,0,1/4,1/4,0,1/4,1/4,1/3,0,0,1/3,0,1/3,0,1/3,0,1/3,1/ 
3,0), nrow=6, ncol=6, byrow=TRUE) 


#setting counter 
nsec<- 0 


#estimating expected number of seconds 
p<- matrix(NA, nrow=172, ncol=6) 
pli,i<- <0, 0, 0, 1, 0,0) 


for (i in 2:172) { 
pli,l<- pli-1,1]%*/tm 
nsec<- nsect+(i-1)*(pli,1]-p[i-1,1]) 
} 


print (nsec) 
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9.967213 


Hence, the mouse spends, on average, 9.967213 seconds in the maze, making 
that many transitions between the states (and an Ezit). 
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Exercises 


EXERCISE 2.1. Consider a one-dimensional random walk with the transition 
probability p = 0.3. Simulate 10,000 trajectories of length 50 steps and calcu- 
late empirical mean and variance. Are the estimates close to the theoretical 
values? 


EXERCISE 2.2. Consider a symmetric one-dimensional random walk that orig- 
inates at 0. 

(a) Simulate 10,000 trajectories with 1,000 steps each. How many of the tra- 
jectories are at point 0 on the 1,000th step? 

(b) Find the theoretical probability of returning to 0 on the 1,000th step. 
Compare to the empirical value. 


EXERCISE 2.3. Simulate 10,000 trajectories of 1D, 2D, and 3D symmetric ran- 
dom walks that start at the origin and continue for at most 1,000 steps. 

(a) Compute how many of them returned to the origin at least once. Com- 
pare the results for different dimensions. Hint: Terminate a trajectory when 
it returns to the origin. 

(b) Consider only the trajectories that returned to the origin within the 1,000 
steps. Compute the average number of steps it took those trajectories to return 
to the origin. Compare the results for different dimensions. 


EXERCISE 2.4. Simulate 10,000 trajectories of a two-dimensional symmetric 
random walk that starts at the origin and continues for a maximum of 1,000 
steps. 

(a) Estimate the probability of a trajectory ever hitting the vertical barrier 
xz = 30. 

(b) Estimate the average number of steps it takes a trajectory to hit the 
barrier, provided it did hit the barrier within the 1,000 steps. 

(c) Estimate the expected value of the y-coordinate at the time when the 
random walk hits the barrier. What should this value be from the theoretical 
point of view? Hint: deduce from a symmetry argument. 
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EXERCISE 2.5. Simulate 100 trajectories of a two-dimensional symmetric ran- 
dom walk that starts at the origin and continues for 1,000 steps or until it hits 
a barrier. The value of the barrier varies between x = 1 and x = 50. Plot the 
empirical probability of hitting the barrier against the barrier value. Discuss 
the pattern you see. 


EXERCISE 2.6. Simulate 1,000 trajectories of a two-dimensional symmetric 
random walk that starts at the origin and continues until it hits a side of a 
square centered at the origin and having a side length of 20. Estimate the 
average number of steps that it takes the random walk to reach the square. 


EXERCISE 2.7. Suppose a gambler starts with a fortune of $7 and will move 
up $1 with probability p or down $1 with probability q = 1—p until he either 
reaches the fortune of $B or is down to $A. 

(a) Prove that the probability that he reaches $B before $A is 


(q/p)*=(a/p)' 
P= (a/ey™—(a/0)P if q/pA1, 
ae if g/p = 1. 
Hint: Show that P; solves the recurrence relation P; = pPj41 + gPi_-1, with 


the border constraints P4 = 0 and Pg = 1. Look for the solution in the form 
P; = c(q/p)' + d, if q/p #1, and P; =ci+d if q/p=1. 


(b) Show that the expected number of games the gambler plays until he 
reaches $B or $A is 


B-A (a/p)'-(@/p)®>_ Bi; 
E,; = ¢ 9? (4/p)*—(a/p)P ap? if q/p # I, 
(B-i)(i— A), if q/p = 1. 
Hint: Show that E; satisfies the recurrence relation E; = pE;4, + qEj;-1 +1 


with the boundary conditions E, = Eg = 0. Look for solutions in the form 
E; = c(q/p)' +d +i/(q—p), if ¢/p #1, and E; = ci+d—7?, if g/p=1. 


(c) Suppose a gambler comes to a casino with $40 and plays a rigged game 
with p = 0.47 until he doubles the amount or is down to $10 (to pay for 
a taxi). Calculate the probability that he walks out of the casino with $80. 
How many games, on average, will he play? Simulate 10,000 trajectories and 
estimate the probability and the expected length of play. 


EXERCISE 2.8. A student visits an Ancient History museum that is open be- 
tween 9AM and 6PM. He enters the museum at 9AM and wanders the rooms 
in a random-walk fashion, spending 30 minutes in each room, and then choos- 
ing a door at random. The museum floor plan is given in the picture. How 
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long will the student spend in the museum, on average? Does he expect to 
leave the museum before it closes for the day? Write down the formula and 
use R to calculate the result. 


Taylor & Francis 
Taylor & Francis Group 


http://taylorandfrancis.com 
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Poisson Process 


3.1 Definition and Must-Know Facts About Poisson 
Process 


A stochastic process {N(t), t > 0} is called a counting process if N(t) gives 
the total number of events occurring by time t. 

A counting process {N(t), t > 0} is said to have independent increments if the 
number of events that occur in non-overlapping time intervals are independent. 
For example, N(5) — N(0), the number of events occurring between times 0 
and 5, is independent of N(10)— (5), the number of events occurring between 
times 5 and 10. 

A counting process {NV (t), t > 0} is said to have stationary increments if the 
distribution of the number of events that occur in any time interval depends 
only on the length of the interval. In other words, N(t)— N(0) and N(t+<s)— 
N(s) have the same distribution that depends only on ¢ and not on s. 

A counting process {N(t), t > 0} is called a Poisson process! with rate A, if: 
(i) no events occur at time 0, i-e., N(0) = 0, (ii) it has independent increments, 
(iii) it has stationary increments, and (iv) P(N(t) = n) = an gore 
0,1,2,.... Note that E(NV(t)) = Var(N(t)) = At. 


9 n= 


Since the rate \ of a Poisson process is a constant not depending on time ¢, 
the process is sometimes referred to as a homogeneous (or stationary) Poisson 
process. 


An interarrival time is the time between two consecutive occurrences of events. 
The interarrival time between (n — 1)st and nth occurrences will be denoted 
by T,, n = 2,3,.... The time of the first occurrence will be denoted by 7}. 


1The reference to a Poisson process first appeared in two independent publications in 
1940. The first was the article by Feller, W. (1940). “On the integro-differential equations of 
purely discontinuous Markov processes.” Trans. Am. Math. Society, 48(3): 488 — 515. The 
second was the Ph.D. dissertation by Lundberg, O. (1940). “On random processes and their 
application to sickness and accident statistics.” Uppsala: Almqvist & Wiksell. 
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The waiting time until the nth event (or event time), S, = T,+T2+---+Tn, 
is the time when the nth event occurs. 


PROPOSITION 3.1. Interarrival times T;,,, n = 1,2,..., are independent ex- 
ponentially distributed random variables with the density function fr, (t) = 
Ne**, t>0. 

Proor: Note that P(T; > t) = P(N(t) = 0) = e~*!. Therefore, T, ~ Exp(,). 


Next, conditioning on the value of the first occurrence, and using independence 
and stationarity of increments, we write 


P(T> ae P(T> >t|T, =s)rAe-* ds 
= P(N(t+s)—N(s) =0|N(s) =1)Ae-** ds 
0 


=| PIN(t) =0) Ae ds=e™* f Ae dg, 
0 0 


that is, Tz ~ Ewp(A). More generally, conditioning on the time of the nth event 
occurrence and again using independence and stationarity of increments, we 
obtain 


P(Tai1 >t) = i BE ca ES = De ids 
0 


-[ P(N(t +s) — N(s) =0|N(s) =n) fs, (s) ds 


= P(N(t) = 0) fs,(s) ds =e~** ih fs,(s) ds =e7**, 
0 0 


REMARK 3.1. Recall that exponential is the only continuous distribution that 
possesses the memoryless property, P([ > t+s|T >t) =P(T > s). Ina Pois- 
son process, the increments are independent and stationary and that implies 
that the process renews itself every moment. Therefore, on an intuitive level, 
the interarrival times should possess the memoryless property and thus be 
distributed exponentially. Also, if, say, on average, there are two occurrences 
per hour (A = 2 per hour), the average waiting time between two occurrences 
is half an hour (mean= 1/A = 1/2 hour). 


PROPOSITION 3.2. The waiting time S;, has Gamma(n, 4) distribution. Thus, 

(Sn) = n/X and Var(S,) = n/d*. The density function is of the form 
n on—-1 

fs,,(8) = or CO, SO) 
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PrROooF: We can write S, as the sum of interarrival times, i.e., S, = Ty + 
To +---+ Thy. It is a general fact in the theory of probability that the sum 
of n independent exponentially distributed random variables with parameter 
» has a gamma distribution with parameters n and A. The quickest proof of 
this fact is through the moment generating functions. 


REMARK 3.2. Let’s look at a Poisson process as a special case of a Markov 
chain. The state space of a Poisson process is S = {0,1,2,...}. The process 
jumps from initial state 0 to state 1 with probability 1, then to state 2 with 
probability 1, etc. The jumps are always of size 1, and transitions between 
states are allowed only in the direction of increase. Moreover, a Poisson process 
includes the time component, so it matters how long the process halts between 
jumps. In fact, we know that it halts an exponentially distributed time. This 
type of Markov chain is called a continuous-time Markov chain. 


REMARK 3.3. It is essential to understand that Poisson arrivals occur one 
at a time. The probability of two simultaneous arrivals is zero since the in- 
terarrival time is exponentially distributed and thus the probability of an 
interarrival time being equal to zero is zero. It means that in situations when, 
say, people can potentially arrive in groups, one needs to count not arrivals 
of individual people, but count the arrival of a group as a single event. For 
example, if we model the process of people joining a ticket line in a movie 
theater, we would be likely to see an entire party joining the line, so we would 
count the party as one arrival. 


EXAMPLE 3.1. A Poisson process is used to model occurrences of rare events. 
Here are some instances of Poisson processes: the number of people who enter 
a store or a bank or a restaurant or a gym or a National Park, the number 
of cars that pass a certain intersection, the number of auto accidents on a 
certain stretch of a freeway, the number of births in a hospital, the number 
of meteors in the night sky, or the number of phone calls to a credit card 
customer service. Typically, natural disasters occur according to a Poisson 
process: earthquakes, volcano eruptions, wildfires, etc. Of course, in all the 
above examples, the considered time period should be short enough for the 
rate of occurrence to be constant. 


Some examples of processes that clearly are not governed by a Poisson law 
are events that happen according to a schedule, for example, the arrival of 
buses along a certain route, road closures due to construction work, quarry 
blasts, building demolitions. Also, events that happen in a competing mar- 
ket, where two rival companies might be scheduling two events at the same 
time. For instance, two pharmaceutical companies might simultaneously bring 


64 Poisson Process 


to the market two cardiac medications, or two production companies might 
release two movies on the same day. In addition, some periodic (or seasonal) 
natural phenomena cannot be modeled as a Poisson process, i.e., eruptions 
of Old Faithful geyser in Yellowstone Stone National Park, ocean tides, the 
appearance of sunspots, etc. 


EXAMPLE 3.2. Tour buses arrive at a roadside mall with restaurants, bring- 
ing 50 tourists each. The times that elapse between consecutive arrivals are 
independent and exponentially distributed with mean of 15 minutes. 


(a) On average, buses arrive every 15 minutes, so, for instance, the expected 
waiting time for the fifth bus to arrive is (15)(5) = 75 minutes, or 1 hour and 
15 minutes. Now we put it in our theoretical framework. The rate of arrival of 
the buses is A = 4 per hour. Denote by $5 the time until the 5th bus arrives. 
We know that Ss has a gamma distribution with parameters n = 5 and A = 4. 
Therefore, E(S;) = 5/4 = 1.25 hours, or 1 hour and 15 minutes. We can also 
compute the variance of S5; as Var(S5) = 5/4? = 0.3125 hours squared, and 
the standard deviation as \/Var(55) = 0.3125 = 0.559 hours. 


(b) The total expected number of tourists who are served lunch at the restau- 
rants between, say, 11AM and 1:30PM is found as follows. Within the two and 
a half hours, we expect 10 bus arrivals, each carrying 50 tourists. Therefore, we 
expect a total of (10)(50) = 500 tourists. To write it formally, let N(t) denote 
the number of buses that arrive by time t. We know that N(t) ~ Poisson(4t). 
We are given that t = 2.5 hours, and therefore, the expected total number of 
tourists is (50)E(N(2.5)) = (50)(4)(2.5) = 500. 


EXAMPLE 3.3. Independence and stationarity of increments in a Poisson pro- 
cess allows elegant computations of conditional probabilities. For example, 
consider a Poisson process with rate \ = 2.2. 


(a) Suppose we want to find the conditional probability that there will be 
8 arrivals by time 5 given that there was 1 arrival by time 2. We can ar- 
gue that since at time 2 the process renews itself, we need to compute the 
probability that within the next 3 time periods there will be 7 more arrivals. 
We write P(N(5) = 8|N(2) = 1) = P(N(5) — N(2) = 7|N(2) = 1) = 
{independence} = P(N(5) — N(2) = 7) = {stationarity} = P(N(5 — 2) = 
7) = P(N(3) = 7) = 22" 6-228) = 0.147243. 


(b) Similarly, conditional expectations can be computed if we utilize inde- 
pendence and stationarity of increments. E[N(9)|N(7) = 10] = E[N(9) — 
N(7) | N(7) = 10] + ELN(7) | N(7) = 10] = E[N(9) — N(7)] + 10 = E[N(2)] + 
10 = (2.2)(2) + 10 = 14.4. 
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(c) Let $39 denote the time of occurrence of the 30th event. Independence and 
stationarity of increments helps us again to compute, for instance, the con- 
ditional expectation of S39 given that 8 events occurred in the first 12 time 
periods. We write E[S3o | N(12) = 8] = 12 + Eltime until 22 more events] = 
12 + E[S29] = 12 + 22/2.2 = 12+ 10 = 22. 


PROPOSITION 3.3. Suppose that in a Poisson process {N(t), t > 0} with rate 
A, an event can be either of category 1 with probability p, or of category 2 with 
probability 1—p. Denote by Ni (t) and No(t) the number of events of category 
1 and 2 that occur by time t, respectively. Note that N(t) = Ni(t) + No(t). 
Then, {Ni(t), ¢ > 0} and {No(t), t > 0} are independent Poisson processes 
with respective rates Ap and X(1— p). 


PRooF: Note that by definition, for an observed value of N(t), Ni(t) has a 
binomial distribution with parameters N(t) and p. Respectively, N2(t) has a 
binomial distribution with parameters N(t) and 1 — p. Therefore, the joint 
probability distribution of Ni(t) and N2(t) can be derived as 
P(Ni(t) = N11, Na(t) = nz) 
= P(Ni(t) =, No(t) = ng | N(t) = Nny, + n2) P(N (t) =n, +n) 


= ’ isi reed — p)” (Apr —At 


ny (n; + ng)! ‘i 
_ Ot)" re OU = PIO" vane 
ny! ng! 


Hence, Ni (t) and No(t) are independent Poisson random variables with rates 
Ap and X(1—p), respectively. Independence and stationarity of increments are 
inherited from those of the process {N(t), t > O}. 


REMARK. In the above proposition, the Poisson process {N(t),t > 0} is 
called the superposition of Poisson processes {Nj(t), t > 0} and {No(t), t > 
O}. In turn, {Ni(t), t > 0} and {No(t), t > 0} are called thinned (or splitted) 
Poisson processes. 


EXAMPLE 3.4. Suppose phone calls to a customer service department in a 
credit card company arrive as a Poisson process with rate 3 per minute. Thirty 
percent of the calling customers experience technical difficulties when using 
their credit cards. 
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(a) The probability that during the next 15 minutes there will be 12 phone 
calls from customers who experience technical difficulties is computed as 
follows. We focus only on the customers who experience technical difficul- 
ties. Call this process {Ni(t), t > O}. We know that it is a Poisson pro- 
cess with the rate \p = (3)(0.30) = 0.9 per minute. So, we can write 


P(N, (15) = 12) = (@29G5)" --(0.9)(15) — 9.10488. 


(b) Suppose now we want to calculate the probability that during the next 15 

minutes there will be 40 phone calls from customers, half of whom experience 

technical difficulties. We denote by {N2(t), t > 0} the Poisson process that 

counts only the callers who don’t experience technical difficulties. Its rate 

is A(1 — p) = (3)(0.7) = 2.1. We know that the processes {Ni(t), t > O} 

and {No(t),t > 0} behave independently. Hence, we write P(N,(15) = 
9)(1 


20, No(15) = 20) = P(N (15) = 20) P(N2(15) = 20) = 29US)" .-(0.9)(15) . 
(eyasy e~ (2-1)(15) — (9.0228)(0.00794) = 0.000181. 


TS 


3.2 Simulations in R 


We will present two simulation methods of a trajectory of a Poisson process. 


SIMULATION 3.1. (EXPONENTIAL INTERARRIVALS). When simulating a tra- 
jectory of a Poisson process, we need first to simulate exponentially distributed 
interarrival times. We base our simulations on standard uniform random vari- 
ables and use the inversion of the cumulative distribution function method 
to obtain exponentially distributed random variables: if u~ Unif (0,1), then 
—¥ In(1 — u) is exponential with mean 1/\. Note that since 1 — wu is also 
Unif (0,1), we can simplify the expression for the exponential random vari- 
ables to —< In wu. 


To plot the simulated trajectory, we use the segment () function which takes 
as arguments vectors of left and right endpoints. Below we present the code 
and graph of a trajectory of a Poisson process with rate 2 that stops when it 
makes the 20th jump. 


#specifying parameters 
lambda<- 2 
njumps<- 20 


#defining states 
N<- O:njumps 
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#setting time as vector 
time<- c() 


#setting initial value for time 
time[1]<- 0 


#specifying seed 
set .seed (333422) 


#simulating trajectory 
for (i in 2: (njumps+1)) 
time [i]<- time[i-1]+round((-1/lambda) log (runif (1)) ,2) 


#plotting trajectory 

# type="n" draws empty frame with no graph 
plot(time, N, type="n", xlab="Time", ylab="State", 
panel.first = grid()) 


segments (time[-length(time)], N[-length(time)], 
time[-1]-0.07, N[-length(time)], lwd=2, col="blue") 


points(time, N, pch=20, col="blue") 
points(time[-1], N[-length(time)], pch=1, col="blue") 


State 
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SIMULATION 3.2. (UNIFORM ORDER STATISTICS). First we need to prove 
the theoretical result. Consider a Poisson process {N(t), t > 0}. Given that 
n events occurred by time t, the times at which the events occurred are dis- 
tributed as the order statistics of a uniform distribution on (0, t). 


To show this result, we derive the conditional density of n waiting times 
S1,59,...,5n, using independence and exponential distribution of the inter- 


arrival times 7), 7>,...,7,. We write 


$1,804...,5n (81) 82,---, $n | N(t) =n) 


_ fr, (81) fry (S2 _ 51) So fr, (Sn _ Sn—1)P(Tr41 >t— Sn) 
P(N(t) =n) 
New> #1 \ eTAS27 81) Lee NeW*MSn—Sn-1) e~At-Sn) QJ 
— ew aay — a 


n. 


This gives us the following algorithm to generate trajectories: 


Step 1. Fix ¢ and generate N(t) ~ Poi(At). 

Step 2. Generate N(t) standard uniform random variables U;,...,Un t). 
Step 3. Order Uj,...,Un() im increasing order, obtaining the ordered set 
Vay». Ue): 

Step 4. Multiply the order statistics by t to obtain the set of event times 
Si — Wa); see »Snit) — tUcn(t))- 

Step 5. Define the states of the Poisson process as N(0) = 0,N(Si) = 
Step 6. Plot the states against time. 


The sample code and plot follow. 
#specifying parameters 
t<- 10 


lambda<- 2 


#specifying seed 
set .seed (32114) 


#generating N(t) 
njumps<- rpois(1,lambda*t) 
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#defining states 
N<- O:njumps 


#generating N(t) standard uniforms 
u<- c() 
uli]<- 0 


for(i in 2: (njumps+1)) 
uli]<- runif (1) 


#computing event times 
time<- t*sort(u) 


#plotting trajectory 
plot(time, N, type="n", xlab="Time", ylab="State", 
panel.first = grid()) 


segments (time[-length(time)], N[-length(time)], 
time[-1]-0.07, N[-length(time)], lwd=2, col="blue") 


points(time, N, pch=20, col="blue") 
points(time[-1], N[-length(time)], pch=1, col="blue") 
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3.3 Applications of Poisson Process 


APPLICATION 3.1. In seismology, occurrence of earthquakes is often mod- 
eled according to a Poisson process. We obtain the data from the Southern 
California Earthquake Data Center’s website hitps://service.scedc.caltech.edu/ 
eq-catalogs/date_mag_loc.php. The data are on earthquakes in Southern Cal- 
ifornia with a minimum magnitude of 3.0 that occurred between 2012 and 
2018. We compute the lengths of the interarrival times and remove those 
earthquakes that were registered within three hours of their predecessors (pos- 
sibly aftershocks). Finally, we conducted a chi-squared goodness-of-fit test to 
see if these times follow an exponential distribution. The R code and output 
follow. 


eq.data<- read.csv(file="./earthquakedata2012-2018.csv", 
header=TRUE, sep=",") 


#creating date-time variable 
datetime<- as.POSIXct (paste(as.Date(eq.data$DATE) , 
eq.data$TIME) ) 


#computing lag 
datetime. lag<- c(0,head(datetime, -1)) 


#computing interarrival times (in hours) 
int.time<- (as.numeric (datetime) -as.numeric(datetime.lag) )/ 
3600 


#removing first value 
int.time<- int.time[-1] 


#removing immediate aftershocks (within 3 hours) 
int<- int.time[int.time>3] 


#plotting histogram 
hist(int, main="", col="dark magenta", xlab="Interarrival 
Time") 
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#binning interarrival times 

binned.int<- as.factor(ifelse(int<40,"1", 

ifelse(int>=—40 & int<80,"2",ifelse(int>=80 & int<120,3", 
ifelse(int>=120 & int<160,"4",ifelse(int>=160 & 
int<200,"5", 

ifelse(int>=200 & int<240,"6","7"))))))) 


#computing observed frequencies 
obs<- table(binned. int) 


#estimating mean for exponential distribution 
mean.est<- mean(int) 


#computing expected frequencies 

exp<- c(1:7) 

exp[1]<- length (int) *(1-exp(-40/mean. est) ) 

exp[2]<- length(int) *(exp(-40/mean. est) -exp(-80/mean.est) ) 
exp[3]<- length (int) «(exp (-80/mean. est) -exp(-120/mean. est) ) 
exp[4]<- length(int) *(exp(-120/mean.est) -exp(-160/mean.est) ) 
exp[5]<- length(int) *(exp(-160/mean.est) -exp(-200/mean.est) ) 
exp[6]<- length(int) * (exp (-200/mean.est) -exp(-240/mean.est) ) 
exp[7]<- length(int) *exp(-240/mean.est) 


obs 
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1 2 3 4 5 6 7 
342 178 117 49 39 24 42 


round (exp, 1) 
319.8 190.5 113.5 67.6 40.3 24.0 35.4 


#computing chi-squared statistic 
print (chi.sq<- sum((obs-exp) 2/exp) ) 


8.883823 


#computing p-value 
print (p.value<- 1-pchisq(chi.sq, df=5)) 


0.1137888 


The number of degrees of freedom in this test is calculated as the number of 
bins minus 1 and minus the number of parameters that have to be estimated 
(in this case one mean), so we get that df = 7—1—1=5. The p-value is larger 
than 0.05, indicating that the earthquakes in the given time frame occurred 
according to a Poisson process. 


APPLICATION 3.2. In sports analytics, a Poisson process is used to model the 
process of goal scoring in a game. Consider a team game where players score 
only one point at a time, for instance, ice hockey. Suppose the points scored 
by team A follow a Poisson process {N4(t), t > 0} with rate A4, and points 
scored by team B are governed by a Poisson process {Nag(t), t > 0} with pa- 
rameter Ag. Assuming the two processes are independent, we can derive some 
interesting results. 


(a) The sum of the two independent Poisson processes N(t) = Na(t)+Np(t) 
is the superposition Poisson process with rate 44 + Ag. It represents the pro- 
cess of scoring by either team A or B. If on average, fans wait for time 1/A,4 
for team A to score, respectively, 1/Ag for team B to score, then the officials 
wait, on average, for a shorter period of time 1/(A4 + Ag) for either team to 
score. 


To see how it works with numbers, suppose team A scores, on average, every 10 
minutes, and team B scores every 12 minutes, on average. Then the expected 


waiting time until any team scores is 1/(1/10+1/12) = 120/22 = 5.45 seconds. 


(b) We can find the probability that one team scores ahead of the other team. 
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Denote by T7'4 and Tz the respective interarrival times. We know that 74 and 
Tg are independent and exponentially distributed with means E(Tg) = 1/4 
and E(Tg) = 1/Ag. We write 


P(team A scores before team B) = P(T,4 < Tp) 


= P(Tzg >t Har= f Erne Nie. OEE ae 
[ Pae> osama f A — 
Now switching \4 and Ag, we get 
AB 
P(team B scores before team A) = P(Tz < T4) = ————. 
Aa +AB 
With our numbers, P(team A scores before B) = — 12/22 = 0.545, 


1/10+1/12 
and P(team B scores before A) = 1 — 0.545 = 0.455. 


(c) We can find the probability of a tie at the end of the game, and also the 
probability that team A (team B) wins. 


Let T denote the length of the game. Using independence of the two processes, 
{Na(t), t > 0} and {Na(t), t > 0}, and expressions for the probability mass 
functions, we write 


P(game ties) = S° P(Na(T) =n, Np(T) =n) 


n=0 


= y aT)” ae Oe BT)" eT aBT — e—(AatAn)T s (Aa Ap T?)" 


n! n! 
n=0 


P(team A wins) = P(Na(T) > Ne(T)) 


=S°S0 P(Na(T) =n +k, Na(L) =n) 
n=0 k=1 
So P(Na(T) =n+k)P(Ne(T) = n) 


— e-atdn)T s Oars T?)" y (4 aI, 


n=0 k=1 
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and, switching A4 and Ap, we get 


Send 2\n co k 
, soy o- (AatAB)T (AaABT*) ; (ABT) 
P(team B wins) = e \41T*8 S- |= S- ( ; 
n=0 k=1 
The duration of the playing time in an ice hockey game is T’ = 60 minutes. 
Therefore, we obtain 


P(game ties) = e~ (1/10+1/12)(60) e Se 
n=0 : 


30" 
-11 
= —~ = 0.1166. 
° dX (nl)? 
We calculated the sum numerically in R. The sum converges after 15 terms. 


sum<- 0 
for(n in 0:15) 
sum<- sum+30n/ (factorial (n) )/2 


sum*exp (-11) 
0.1165575 


Further, 
P(team A wins) = e~! 3 [= 3 | = 0.5590 
7 rr n! a (n+k) , 
R code given below computes this double sum numerically. 


sum.n<- 0 
for (nm in 0:15) { 
sum.k<-0 
for (k in 1:15) 
sum.k<- sum.k+6“k/factorial (ntk) 
sum.n<- sum.n + 30‘n/factorial (n) *sum.k 


} 
sum.n*exp(-11) 
0.5589743 


Finally, 


Co n lo e) k 
P(team B wins) =e“! $7 [= Ss | = 0.3244, 
n=0 k=1 : 
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as the R code below computes 


sum.n<- 0 
for (nm in 0:15) { 
sum.k<-0 
for (k in 1:15) 
sum.k<- sum.k+5k/factorial (ntk) 
sum.n<- sum.n + 304n/factorial (n) *sum.k 


a: 


sum.n*exp(-11) 


0.3244495 


Note that the three probabilities add up to 1, as they should be. 


APPLICATION 3.3. One famous application of a Poisson process is that of a 
pedestrian versus traffic flow. A pedestrian needs to get to the other side of a 
road. Assume that cars pass according to a Poisson process with rate 2. 


(a) If it takes the pedestrian time 7 to cross the road, how long, on average, 
will it take the person to get to the other side? 


Note that the person has to wait for a gap in traffic of length at least 7 before 
he/she can cross. Denote by T' the total time (waiting plus crossing). Suppose 
the person just approached the road. The Poisson process renews itself at this 
moment, and thus, the person has to wait an exponential time with a mean 
1/A for the next car. Call this time T;. If T; > 7, then the person can safely 
cross the road and T = T. If, however, T; <7, the person has to wait for T) 
for the first car to pass, and then the process renews itself and the pedestrian 
would have to wait for an additional time T that has the same distribution at 
T. Thus, T can be written as 


pat? if T, > 1, 
~|R+ee, #7, <4, 


where 7} is an exponentially distributed random variable with mean 1/,. Con- 
sequently, 


3(T) = TP(T,; > 7) + | tre! dt + E(T)P(T; <7). 
0) 
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This can be rewritten as 


(T)P(T, > 7) =7TP(M Sr) + | tre~** dt 
0 


From here, 


(f= aman cae — 1). 


Suppose, on average, a car passes every 20 seconds (that is, A = 1/20 = 0.05 
cars per second), and the pedestrian needs t = 30 seconds to cross the road. 
Thus, on average, it takes the person E(T) = ggg (eG — 1) = 69.63 
seconds to cross the road. 


(b) How many cars, on average, will pass by before the pedestrian can cross? 
Let N be the number of cars that pass by before the person can cross. Then 
N > n, if and only if the first n interarrival times are all less than 7. Hence, 
P(N >n)=(1- ee We can compute the probability of N =n as 
P(N =n) =P(N >n)—-P(N>n41)=(1- 7)" —- (1—-e7)™. 


Let a = 1 — e~*7. The expected value of N can then be computed as 


(N) = 5° nP(N =n) = >> n(a™ —a™*1) =a— a? + (2)(a? — a?) 
n=0 n=0 
1 
+G)(@" = 8) 6 Saha? 2a? 0 ie SPS ert 4: 
—a 


Note that E(T) = (1/A)E(NV). It is intuitively so, because the person has to 
wait until an average of E(V) cars pass, and the average waiting time between 
these cars is 1/ seconds. In our numeric example, A = 0.05 and 7 = 30. So, 
u(N) = e(0-05)(30) _ 1 — 3.48 cars. 


rr 


Exercises 


EXERCISE 3.1. Let {N(t), t > 0} be a Poisson process with rate . Find the 
joint probability distribution P(N (s) =m, N(t) =n), for any t > s > 0, and 
n>m> 0. 
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EXERCISE 3.2. Show that for a Poisson process {N(t), t > 0} with rate A, 
the covariance between N(s) and N(t) is equal to A min(s,t), for any s,t > 0. 


EXERCISE 3.3. An insurance agent handles policyholders’ claims. Claims are 
submitted on weekdays according to a Poisson process with a rate \ = 5 per 
day. 

(a) If there were two claims submitted on Monday and three on Tuesday, 
what is the probability that by the end of the day on Friday there will be a 
total of 16 claims submitted that week? 

(b) In the new calendar year, the agent opens for business on Monday, Jan- 
uary 2. On what day does he expect to see the 100th claim? 


EXERCISE 3.4. A salesperson contacts customers over the phone and offers 
his product. Assume that the times that pass between consecutive phone calls 
(that includes the call and the break in-between) are independent and expo- 
nentially distributed with mean of 5 minutes. He estimates that 15% of all the 
customers he calls actually buy his product. 

(a) Calculate the expected number of successful sales in the next 2 hours. 
(b) Compute the probability that within 1 hour he places 15 calls, 5 of which 
result in a sale. 

(c) Find the conditional probability that he makes 10 sales in 4 hours, given 
that he has made 8 sales the first hour. 


EXERCISE 3.5. People contract a disease according to a Poisson process with 
an unknown rate \. Suppose the incubation period until symptoms of the dis- 
ease show is a random variable with a known cumulative distribution function 
F’. Let N,(¢) denote the number of individuals who have shown symptoms by 
time t, and let No(t) be the number of individuals who have not yet shown 
any symptoms by time t. 

(a) Argue that {Ni(t), > 0} and {No(t), t > 0} are independent Poisson 
processes with means 


M(t) => f F(u)du and (Na() => | (1 — F(u)) du. 


(b) For a known time ¢ and observed number of individuals showing symp- 
toms E(Nj(t)), prove that the estimated number of individuals infected but 
not yet showing symptoms is 


zs t 
(Ni (t)) fi = F(w)) du 
if F(u) du 
(c) Suppose the incubation period until symptoms show is an exponentially 
distributed random variable with a mean of 2 days. If 1,000 individuals show 


E(N2(t)) = 
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symptoms of a disease by day 10, estimate the number of individuals who are 
also infected but haven’t shown the symptoms yet. 


EXERCISE 3.6. Areas of high road surface distress (potholes or cracks) that 
need immediate attention of road maintenance operators are distributed ac- 
cording to a Poisson law with a rate of 2.8 per mile. 

(a) What is the average number of distressed road surface areas on a 10-mile 
stretch of a freeway? 

(b) Simulate locations of 30 distressed surface areas. What is the total length 
of the road in your simulation? 

(c) Suppose there are 30 distressed surface areas on a 10-mile stretch of a 
freeway. Simulate locations of those areas. 


EXERCISE 3.7. The National Geophysical Data Center’s website 
https://www.ngdc.noaa.gov/hazel/view/hazards/volcano/event-search/ 


provides access to the Global Significant Volcanic Eruptions Database. Verify 
that those volcanic eruptions in the past 100 years can be modeled as a Poisson 
process. 


EXERCISE 3.8. Two teams are playing basketball. Team A opportunities to 
score appear as a Poisson process with a rate of 0.5 per minute. Suppose that 
25% bring one point, 40% bring two points, 20% bring the team three points, 
and the others result in missed shots. For team B, the opportunities come 
as a Poisson process with a rate of 0.4 per minute, of which 25% result in a 
1-pointer, 50% result in a 2-pointer, 15% result in a 3-pointer, and the rest 
are missed. 

(a) How long, on average, does the stadium have to wait until a team scores? 
(b) How long, on average, will the fans wait until team A scores? Team B 
scores? 

(c) What is the probability that team A scores before team B? Team B scores 
before team A? 

(d) What is the probability that at the end of the 48-minute game, teams A 
and B will score the same number of 1-pointers, the same number of 2-pointers, 
and the same number of 3-pointers? 


EXERCISE 3.9. In a popular nursery rhyme, 


Itsy bitsy spider went up the water spout. 

Down came the rain and washed the spider out. 
Out came the sun and dried up all the rain, 

And the itsy bitsy spider went up the spout again. 
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Assume that the length of the downspout is 30 feet, and the spider climbs 
with a constant speed of 1 foot per minute. The rain comes down as a Poisson 
process with a rate of 2 per hour. 

(a) Find the expected time it takes the spider to reach the top. 

(b) Find the expected number of times the spider will be washed down before 
it reaches the top. 
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Nonhomogeneous Poisson Process 


4.1 Definition of Nonhomogeneous Poisson Process 


The Poisson process considered in the previous chapter has a constant rate 
X. In some situations, it is difficult to assume that the rate doesn’t change 
over a large period of time. In this case, we can make 4 depend on time t and 
define a Poisson process with rate A(t). The rate is now called the intensity 
rate or intensity function. The process still starts at zero at time zero, and its 
increments are still independent, but now the increments are non-stationary. 


A counting process {N(t),t > O} is called a nonhomogeneous (or non- 
stationary, or time-dependent) Poisson process? if: (i) N(0) = 0, (ii) incre- 
ments are independent, and (iii) for all s,t > 0, 


ies \(u) du)" (ae A(u) an) 


ae ae n> 0. 


| — 


n! 
t 
Define the function A(t) = | (uw) du. It is called the integrated intensity 


rate function or the mean value function. The probability mass function of a 
nonhomogeneous Poisson process can be written in terms of A(t) as 


P(N(t+s) —N(s) =n) = a eo [A(e+8)-A(s) | eo 


Note that for s,t > 0, E(N(¢+ s) — N(s)) = A(t+ 8) — A(s). 


REMARK 4.1. Here we formulate an alternative definition of a nonhomoge- 
neous Poisson process. It can be shown that the two definitions are equivalent. 
We will need this definition to justify the method we use in Simulation 4.3. 
A counting process {N(t), t > 0} is termed a nonhomogeneous Poisson pro- 
cess with the intensity rate A(t), t > 0, if: (i) N(O) = 0, (ii) increments are 


1Introduced by a prominent statistician Sir David Roxbee Cox in his 1955 paper “Some 
Statistical Methods Connected with Series of Events.” Journal of the Royal Statistical So- 
ciety. Series B (Methodological), 17(2): 129 — 164. 


DOI: 10.1201/9781003244288-4 81 
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independent and stationary, (iii) for any fixed t > 0 and any small positive 
increment At, P(no events happen in [¢,¢+At]) = 1— (At) +0(At), and (iv) 
P(one event happens in [¢,t+ At]) = A(At) + o(At), where o(At) (pronounced 
“little oh of delta t”) denotes any function of At that goes to zero faster than 


At. That is, o(At) = { f (At) : limatso At) _ o}. 


EXAMPLE 4.1. Throughout the day, arrivals of phone calls to a doctor’s office 
can be modeled as a nonhomogeneous Poisson process with the intensity rate 


10, if9AM <t < 10:30AM, 
5, if 10:30AM <t < 12PM, 
8, if 12PM <t< 1PM, 

4, if 1PM <t< 5PM. 


\t) = 


9 


(a) To calculate the integrated rate function A(t), we need to define the time 
variable as ranging between 0 and 8 hours of the workday. We can rewrite the 
intensity rate as 

10, if0<t< 1.5, 

5, if 1b5<t<3, 

8 if38<t<4, 

if4<t<8. 


A(t) = 


The integrated rate function is then computed as 


Jo 10 du = 108, if0<t<155, 
15+ ff, 5du=15+5(—-1.5), f15<¢<3, 
22.54 fy 8du=2254+8(t-3), if3<t<4, 
30.5+ f; 4du=30.54+4(t—4), if4<t<8. 


A(t) =| A(u) du = 


(b) Below we plot both functions. The R codes are provided. 


#plotting intensity rate 

t=c(0,1.5,3,4,8) 

lambda=c(10, 10,5,8,4) 

plot(t, lambda, type="n", col="blue ", xlim=c(0,8), 
ylim=c(0,12), xlab="Time", ylab="Intensity rate") 

segments (t[-5]+0.07, lambda[-1], t[-1], lambda[-1], lwd=2, 
col="blue") 

points(t, lambda, cex=1.2, pch=19, col="blue") 
points(t[-5], lambda[-1], cex=1.2, pch=1, col="blue") 
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#plotting integrated rate function 

t<- c(O, 1.5, 3, 4, 8) 

Lambda<- c(0, 15, 22.5, 30.5, 46.5) 

plot(t,Lambda, type="1", lwd=2, col="blue", xlim=c(0,8), 
ylim=c(0,50), xlab="time", ylab="integrated rate function") 
points(t, Lambda, cex=1.2, pch=16, col="blue") 
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(c) Suppose we want to compute the probability that there will be 15 phone 
calls between 11AM and 2PM. The time 11AM corresponds to 2 hours and 
2PM corresponds to 5 hours after the office opens. We write 


A(5) — A(2)]"° 
P(N(5) — N(2) = 15) = [A(5) = ) - [AG)-AQ)] 
15 
_ (80.5 + 45 ~ 4) ~ 05 + 5(2=1.5))) ° - (30.544(6-4)- (1545(2-1.5))) 
7 15! 
1 15 
= oo e~*” = 0.011468. 


(d) Finally, we want to compute the average number of phone calls per day. 
We compute 


5(N(8) — N(0)) = A(8) — A(O) = 30.5 + 4(8 — 4) — 0 = 46.5 calls. 


rr 


4.2 Simulations in R 


Recall that in Section 3.2 we discussed two simulation methods for trajectories 
of a homogeneous Poisson process. In the present section, we generalize these 
methods to the case of a nonhomogeneous Poisson process. 


SIMULATION 4.1. (EXPONENTIAL INTERARRIVALS). In this method, we sim- 
ulate interarrival times. In the nonhomogeneous case, the distributions of in- 
terarrival times are not independent. They are obtained as follows. 


The first interarrival time has the cumulative distribution function Fy, (t) = 
1— eh), t > 0. For a fixed time of the first event occurrence S$; = T; = 51, 
the cumulative distribution function of T) is 


Frajs, (tls1) = 1 — e MH) -A00), 4 > 0, 
In general, for a given waiting time for the nth event S,, = s,, the conditional 
distribution of the interarrival time T;,,1 is 


Fr. 8,(t| $n) =1 — e7 Mtt)-AGD) £0, n> 1. 


+1 | Sn 
As an example, here we give the code that simulates a trajectory of a 
nonhomogeneous Poisson process with the integrated rate function A(t) = 
t+ 0.05t?, ¢ > 0. In the code we first generate standard uniform random 
variables U;,z2 = 1,...,n, and then compute event times by inverting the 
cumulative distribution function. We write 1 — e~(S:+0.05Si) — U,, which 
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solution is S; = ,/100 — 20 In(1 — U1) — 10. It can be simplified by replacing 


1—U, by U, since both are standard uniform random variables. Thus we have 
Sy, = ,/100 — 20 In(U;) — 10. 

The second event time 52 solves 1 — e = Us, or equiv- 
alently, (S2 +10)? — ($; +10)? = —20In(1 — U2). The solution is (with 1— Uz 
replaced by U2) Sz = ./(Si1 + 10)? — 20 In(U2z) — 10. The general recurrence 
formula is Sy41 = (Sn, +10)? — 20 In(U,+1) — 10. We continue generating 
the event times according to this formula until we reach a pre-specified num- 
ber of events of the process. The code and graph are given below. 


= [52+0.05 S3—(S1+0.05 $?)] 


#specifying parameters 
njumps<- 20 


#defining states 
N<- O:njumps 


#defining times as vectors 
time<- c() 


#specifying seed 
set .seed (76855) 


#generating standard uniforms 
u<- c() 

for(i in 1:njumps) 

uli]<- runif (1) 


#computing event times 
time[1]<- 0 
time [2]<- sqrt (100-20*log(u[1]))-10 


for(i in 3:(njumps+1)) { 
time [i]<- sqrt ( (time [i-1]+10)%2-20*log(u[i-1]))-10 
} 


#plotting trajectory 
plot(time, N, type="n", xlab="Time", ylab="State", 
panel.first=grid()) 


segments (time[-length(time)], N[-length(time)], 
time[-1]-0.07, N[-length(time)], lwd=2, col="blue") 


points(time, N, pch=20, col="blue") 
points(time[-1], N[-length(time)], pch=1, col="blue") 
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SIMULATION 4.2. (UNIFORM ORDER STATISTICS). Let {N(t), t > 0} denote 
a nonhomogeneous Poisson process with the intensity rate A(t), t > 0, and 
integrated rate function A(t), t > 0. Given that n events occurred by time t, 
the conditional joint density of n waiting times 5), S2,...,5,,, is derived as 


F882 008m (St, Bay «0+48n | NG) =n) 


_ fo, (81) fr, (82 = $1) +++ + ft, (Sn = 8n-1) Pn > t = Sn) 
P(N(t) =n) 


A(s1) e~A(s1) A(s2 — $1) e7 (A(s2)-A(s1)) seerace N(Sn = Sn—1) 
e—(M8n)—A(sn—1)) @—(A(t)—A(Sn)) 


This means that the event times are order statistics from the distribution 


nN ee eee 
with density fs(s) = oe 0 <s <t, and the cumulative distribution func- 


A(t)’ 
A(s) 


tion F'g(s) = GW O<s<t. 


In our example with A(t) = ¢ + 0.05#?, t > 0, to generate a trajectory, we 
proceed as follows. 
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Step 1. Fix t and generate N(t) ~ Poi(A(t)). For instance, for t = 10, 
A(10) = 10 + (0.05)(107) = 15, and so, we would generate N(t) ~ Poi(15). 
Step 2. Generate N(t) standard uniform random variables U;,...,Un(). 
Step 3. Order U1,...,Uy (4) in increasing order, obtaining the ordered set 
Ui); obsats Unt)): 

Step 4. Compute waiting times 5},...,Sy() that are positive solutions of 
A(Si) 
A(t) 
10,/1 + 30) — 10. 
Step 5. Define the states of the Poisson process as N(0) = 0,N(Si) = 
1, N(S2) = 2,..., N(Swy) = N(t). 

Step 6. Plot the states against time. 


the equations = Ug, or S; + 0.05 oF = 16 Ui). That is, 5; = 


The code and plot follow. 


#specifying parameters 
t<- 10 
Lambda<- t+0.05*t2 


#specifying seed 
set .seed (997755) 


#generating N(t) 
njumps<- rpois(1, Lambda) 


#defining states 
N<- O:njumps 


#generating N(t) standard uniforms 
usc) 
uli]<- 0 


for(i in 2: (njumpst1)) 
uli]<- runif (1) 


#computing event times 
time<- 10*sqrt(1+3*sort(u))-10 


#plotting trajectory 
plot(time, N, type="n", xlab="Time", ylab="State", 
panel.first=grid()) 
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segments (time[-length(time)], N[-length(time)], 
time[-1]-0.07, N[-length(time)], lwd=2, col="blue") 


points(time, N, pch=20, col="blue") 
points(time[-1], N[-length(time)], pch=1, col="blue") 
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Below we introduce a third method of simulating a trajectory of a nonhomo- 
geneous Poisson process. In the case of a homogeneous Poisson process, this 
method becomes trivial and hence not useful. 


SIMULATION 4.3. (THINNING). Let {N(t), t > 0} denote a nonhomogeneous 
Poisson process with the intensity rate A(t), t > 0, and suppose its integrated 
rate function doesn’t have an explicit form or is not easily invertible, so the 
previous two simulation methods won’t work for this process. Here we in- 
troduce another method, called the thinning method. In this method, first 
we generate a nonhomogeneous process {N*(t), t > 0} with intensity rate 
A*(t), t > 0, that uniformly dominates A(t). That is, A(t) < A*(t) for all t > 0. 
The process is selected in such a way that its integrated rate function is in- 
vertible, and so N*(t) can be generated by either of the previous two methods. 
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Sometimes A*(t) may be chosen to be a constant, resulting in a homogeneous 
Poisson process, which is even simpler to generate (Section 3.2). 


Further, once the event times sj,...,si; for the process {N*(t), t > O} are 
generated, they are “thinned” according to the following acceptance-rejection 
rule: if a standard uniform random variable is less than or equal to the ratio 
A(s*)/A*(s*), the time s* is accepted, otherwise, rejected. 


The accepted times are the event times for the process {NV (t), t > 0}. To show 
this, we use the alternative definition of a nonhomogeneous Poisson process 
given in Remark 4.1 and argue as follows. The process {N*(t), t > 0} starts 
at zero and has independent and stationary increments. These properties are 
inherited by {N(t), ¢ > 0}. Further, in an infinitesimally small interval of 
length At, there are zero occurrences of the process {NV (t), t > 0} if there are 
no occurrences of the process {N*(t), t > 0} or there is one occurrence but it 
is rejected. There is one occurrence of the process { N(t), t > 0} if there is one 
occurrence of the process {N*(t), t > 0} and it is accepted. Denoting by T* 
an interarrival time, and by U a standard uniform random variable, we write 


(At) ) 


P(N(At) = 0) = P(N*(At) = 0) + P(N*(At) = 1) P(U > \* (At) 


(At) ) 


= Pr > At) + P(T™* < At) P(U a (At) 


Ae a (At) 
_ —A*(At) d* (At) 
“= (1 (1 \* an) 


= 1 = d*(At) + of At) + (X*(At) + o(A8)) (1 - an) 


= 1-— (At) + o0(At), for small At, 


and 


P(N(At) = 1) = P(N*(At) = 1) P(U < 


= ¥(@s) 
; At) —ascan) MAM) 
=P(I* < A) P(U Ss Toy) = (2 - eC) Sos 


= (A*(At) + o(At)) Sr = \(At) + o(At), for small At. 


The above derivation shows that N(t) is a nonhomogeneous Poisson process 


with the intensity rate function X(t). 


Below we present the code that simulates a trajectory of a nonhomogeneous 
Poisson process with the intensity rate A(t) = 20 + 20sin(zt), t > 0. For this 
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process, the integrated intensity rate function A(t) = 20t— #8 cos(mt) +78, t > 
0, is not readily invertible. However, A(t) < 40, and so we simulate event times 
for a Poisson process with rate \*(t) = 40, and then accept an event time s if 
U < X(s)/A*(s) = 20(1 + sin(7s))/40 = 0.5(1+sin(7s)), and reject otherwise. 


#specifying parameters 

lambda<- function(t) { 20+20*sin(pi*t) } 
lambda.star<- function(t) 40 
Lambda.star<- function(t) 40*t 


#specifying seed 
set .seed (2866514) 


#generating N(10) 
njumps<- rpois(1, Lambda.star(10)) 


#generating N(10) standard uniforms 
u<-c() 
ul1]<- 0 


for(i in 2: (ajumps+1)) 
uli]<- runif (1) 


#computing event times 
time.star<- 10*sort(u) 


#thinning event times 
accepted<- c() 

time<- c() 
accepted[1]<- 1 
time[1]<- 0 


for (i in 2:(njumps+1)) { 

if (runif(1)<= lambda(time.star[i])/lambda.star(time.star[i])) 
accepted[i]=1 else accepted[i]=0 

} 


time<- time.star[-which(accepted==0) ] 
N<- 0: (length(time) -1) 


#plotting trajectory 
plot(time, N, type="n", xlab="Time", ylab="State", 
panel.first = grid()) 
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segments (time[-length(time)] ,N[-length(time)], time[-1]-0.07, 
N[-length(time)], lwd=2, col="blue") 


points(time, N, ylim=c(0,120), pch=20, col="blue") 
points (time[-1] ,N[-length(time)],pch=1, col="blue") 


100 + 


80 + 


40 + 


20 5 


rr 


4.3. Applications of Nonhomogeneous Poisson Process 


APPLICATION 4.1. In Application 3.1, we modeled the occurrence of earth- 
quakes in Southern California between 2012 and 2018 via a Poisson model and 
concluded that it fits the data well. Now we will try to fit a nonhomogeneous 
Poisson process to a larger data set covering the time span between 2010 and 
2020. In this larger time interval, the event rate doesn’t stay constant. 


First, we estimate the intensity rate function. The code below calculates \ 
as the ratio of the number of earthquakes per month over the number of 
days in that month, producing the estimated daily rate for each of the 120 
months. One outlying value is removed (for July of 2019), leaving us with 119 
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values. As the time variable, we compute the accrued number of days from 
09/02/2010 to the median day for each of the 119 months. Next, we plot the 
estimated A against time and fit a fourth-degree polynomial regression. We 
then define the intensity rate function A(t) as the fourth-degree polynomial 
with the estimated coefficients. After that, we subdivide the timeline between 
09/02/2010 and 08/28/2020 into 9 bins of size 400 days and calculate the 
observed number of earthquakes in each bin. Then we compute the expected 
number of occurrences in each bin as the integral of A(t) between the lower 
and upper values of time in this bin. Finally, we compute the chi-squared 
statistic for the goodness-of-fit test and output the p-value. 


The code and all the necessary outputs follow. 


eq.data<- read.csv(file="./earthquakedata2010-2020.csv", 
header=TRUE, sep=",") 


#creating date-time variable 
eq.data$datetime<- as.POSIXct (paste(as.Date(eq.data$DATE) , 
eq.data$TIME) ) 


#computing lag 

eq.data$datetime.lag<- c(0,head(eq.data$datetime, -1)) 
#removing first row 

eq.data<-eq.data[-1,] 


#computing interarrival times (in hours) 
eq.data$elapsed.time<- (as.numeric(eq.data$datetime) 
-as.numeric(eq.data$datetime.lag))/3600 


#removing immediate aftershocks (within 1 hour) 
eq.data<- eq.dataleq.data$elapsed.time>1,] 


#creating year-month variable 
eq.data$year.month<- format(as.Date(eq.data$DATE), "/“Y-%m" 


#creating unique year-month and number of earthquakes per 
month 

freq.month<- data.frame(table(eq.data$year.month) ) 

year .month. unique<-freq.month[, 1] 

neq.month<- freq.month[,2] 

neq.month 
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[1] 37 43 29 35 29 19 19 26 18 23 24 11 12 16 19 6 
[17] 10 15 10 16 19 12 20 27 14 12 8 10 
[33] 20 14 7 14 14 12 
[49] 10 14 5 9 5 10 
[65] 10 13 7 8 4 413 
[81] 9 10 10 8 8 8 9 9 5 6 it 
[97] 11 10 9 7 14 114 9 5 18126 35 20 11 25 21 

[113] 15 12 12 16 19 30 10 15 


11 3 10 


Wana nN N 
i 
on 
BPR 
NB 

o 
ww 
N 
ODMDNN O 
o 
on 
—_ 
(o>) 


#removing outlier (July, 2019, 107th entry) 
year.month.unique<- year.month.unique[-107] 
neq.month<- neq.month[-107] 


#computing number of days per month 

library (lubridate) 

day1.month <- ymd(paste(year.month.unique,"01", sep="-") ) 
library (Hmisc) 

ndays.month<- monthDays(as.Date(day1.month, "/%Y-/m-/d") ) 


#estimating intensity rate of earthquakes per day 
lambda<- neq.month/ndays.month 


#computing cumulative number of days until median day of each 
month 

median.time<- c() 

ndays.total<- c() 

median.time[1]<- ndays.month[1]/2 

ndays.total[1]<- ndays.month[1] 


for (i in 2:length(ndays.month)) { 

median.time[i]<- ndays.total[i-1] + ndays.month[i]/2 
ndays.total[i]<- ndays.total[i-1] + ndays.month[i] 

} 


#plotting lambda against median time 
plot (median.time, lambda) 
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#regressing lambda on time 


median.time 
median.time 
median.time 


.re<- median.time/1000 
.sq<- median.time.re’2 
.cu<- median.time.re’3 


median.time.qd<- median.time.re”4 
glm(lambda ~ median.time.re + median.time.sq + median.time.cu 


+ median.time.qd) 


Coefficients: 
(Intercept) 
1.11674 


median.time.re median.time.sq median.time.cu 
-1.75263 1.41060 -0.49950 


median.time.qd 
0.06504 


#adding fitted line 
lambda.fn<- function(t) { 1.11674-1.75263*(t/1000)+1.41060* 
(t/1000)%2 -0.49950*(t/1000)*3+0.06504*(t/1000)%4 } 


lines(median.time, lambda.fn(median.time), lwd=2, 
col="blue") 
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#conducting goodness-of-fit test 


#binning times 

time. binned<- as.factor(ifelse(as.Date(eq.data$DATE) 
<"2011/10/07","1", ifelse(as.Date(eq.data$DATE) >="2011/10/07"& 
as .Date(eq.data$DATE) 

<"2012/11/10","2", ifelse(as.Date(eq.data$DATE) >="2012/11/10" 
& as.Date(eq.data$DATE) <"2013/12/15","3", 
ifelse(as.Date(eq.data$DATE) 

>="2013/12/15"& as.Date(eq. data$DATE) <"2015/01/19","4", 
ifelse(as.Date(eq.data$DATE) >="2015/01/19"& 

as .Date(eq.data$DATE) 

<"2016/02/23","5", ifelse(as.Date(eq.data$DATE) >="2016/02/23"& 
as .Date(eq.data$DATE) <"2017/03/29","6", 
ifelse(as.Date(eq.data$DATE) 

>="2017/03/29"% as.Date(eq. data$DATE) <"2018/05/03","7", 
ifelse(as.Date(eq.data$DATE) >="2018/05/03"& 

as .Date(eq.data$DATE) 

<"2019/06/07", "8", "9"))))))))) 


#computing observed frequencies 
obs<- table((time.binned) ) 
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#computing expected frequencies 

exp<- c() 

exp[1]<- integrate(lambda.fn, 0, 400)$value 
exp[2]<- integrate(lambda.fn, 400, 800)$value 
exp[3]<- integrate(lambda.fn, 800, 1200)$value 
exp[4]<- integrate(lambda.fn, 1200, 1600)$value 
exp[5]<- integrate(lambda.fn, 1600, 2000)$value 
exp[6]<- integrate(lambda.fn, 2000, 2400)$value 
exp[7]<- integrate(lambda.fn, 2400, 2800)$value 
exp[8]<- integrate(lambda.fn, 2800, 3200) $value 
exp[9]<- sum(obs)-sum(exp) 


obs 


1 2 3 4 5 6 %7 8 9 
326 198 139 141 108 112 113 121 376 


round (exp, 1) 
333.5 192.9 137.7 123.2 120.7 117.3 116.2 136.7 355.7 


#computing chi-squared statistic 
print (chi.sq<- sum((obs-exp)2/exp) ) 


7.494979 


#computing p-value 
print (p.value<- 1-pchisq(chi.sq, df=3)) 


0.05768759 


Note that we estimated five parameters in the polynomial regression, there- 
fore, we needed to pick at least seven bins to have a non-degenerate number 
of degrees of freedom for the test. The total time span in the data set is 3,647 
days (without the outlier), so it was reasonable to divide the range into 9 bins 
of size 400 days each (the last bin has 447 days). Nine bins result in 3 degrees 
of freedom, (df = 9-—1-—5=83). 


Looking at the chi-squared statistic and the corresponding p-value, we can 
conclude at the 5% level of significance that the mean values in this process 
are well modeled by the fitted integrated intensity rate function. 


It remains to show that the interarrival times have an exponential distribution. 
It is not possible to do it rigorously because of the nonhomogeneous nature 
of the process, but at least we can construct histograms for the interarrival 
times in each of the nine bins to see that they have the shape of an exponential 
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density. As an illustration, below we present the histograms for the first and 
the last bins. 


inti<- eq.data$elapsed.time[as .Date(eq.data$DATE) 
<"2011/10/07"] 
hist(int1, main="", xlab="", ylab="", col="light blue") 


. ii T T T T 1 
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int9<- eq.data$elapsed.time[as .Date(eq.data$DATE) 
>="2019/06/07"] 
hist(int9, main="", xlab="", ylab="", col="light blue") 


300 4 
250 + 
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APPLICATION 4.2. Reliability engineers are concerned with the ability of man- 
ufactured systems or components to function without failure. Once failed, the 
item is repaired (in a repairable system) or replaced (in a non-repairable sys- 
tem). A stochastic model of the number of failures that has been widely used 
in practice by reliability engineers is a nonhomogeneous Poisson process with 
the power-law intensity rate (or repair rate) \(t) = a8 t?-!, a,B>0, t>0. 
This function is very flexible because it models increasing rates if 8 > 1, or 
decreasing rates if 0 < 6 < 1. If 6 = 2, the failure rate function degenerates 
into A(t) = 2at, which corresponds to the homogeneous Poisson process. 
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(a) Let us study this model. Denote by {N(t), ¢ > 0} the number of fail- 
ures by time t. The integrated intensity function is A(t) = i ab ub) du = 
at®, a,8 >0,t > 0. The probability mass function of N(t) has the form 


(at? _ as)" 


B 
en alts") b> 5 >, 
TN 


P(N(t) — N(s) =n) = 


Note that the expected value of N(t) is E(N(t)) = A(t) = at®. 


An important question in reliability analysis is how to estimate a and $ from 
the data. We address this question here. Two methods are commonly used to 
estimate the parameters. The first uses the linear regression approach, whereas 
the second one produces the maximum likelihood estimator. 


METHOD 1 (LINEAR REGRESSION). Since E(N(t)) = at, we can state the 
empirical analog N(t) = at’, or, equivalently, In(N(t)) =Ina@+ @ Int. Thus, 
In(V(t)) can be regressed linearly on Int to obtain the estimated intercept 


In@ and slope £. 


To look at a numeric example, assume that S;,, k = 1,2,...,30, the times to 
failure (in weeks) of certain auto parts during the pilot testing period, are as 
given in the table below. The second variable is N(S;) = k, the total number 
of failures up to and including time Sx. 


2.36 10.16 11 19.70 21 


il 

2 

3 

4 

5 

6 

7 

8 

9 
10 


Now we regress In(N(S;,)) = In(k) on In($;) to obtain @ = e~°-6854 = 0.5039, 
and 8 = 1.2570. The code and output are below. 


reliability.data<- read.csv(file="./reliabilitydata.csv", 
header=TRUE, sep=",") 


x<- log(reliability.data$time) 
y<- log(reliability.data$nfailures) 


gim(y ~x) 
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Coefficients: 
(Intercept) x 
-0.6854 1.2570 


plot(x,y, xlab="1n(time)", ylab="1n(nfailures)") 
lines(x, -0.6854+1.2570*x) 
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Using the known formulas for the estimators of the slope and intercept in 
linear regression, we write the explicit expressions for G and @. Here N is the 
total observed number of failures. In our example, N = 30. We have 


4g _ Dhes Lint) n(Si)} — [inv /] [kes (S| 
Sh, [mis]? - @/ny [OX m(syp 


and 


N 
a@ = exp {(1/N) [m(v}) — B S* In(S;)] } 


> 
Il 
a 
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We can verify in R that these expressions give us the same results as above. 
Indeed, 


print (betal.hat<- (sum(x*y)-log (factorial (N) )*mean(x))/ 
(sum(x*2) -N*(mean(x)) %2)) 


1.256956 


print (alphal.hat<- exp(log(factorial(N))/N-beta1.hat*mean(x) )) 


0.5038849 


Before we go to the second method of estimation, we need to define the distri- 
bution of the failure times S;,k = 1,2,.... Given that the kth failure occurred 
at time s,, the distribution of S;4, is Weibull with the scale parameter a and 
the shape parameter 3. The density is 


B 
J Sp¢4 |S (t | Sk) = oa: BtP-? e° (+?—sf)_ t> Sr, 


and the cumulative distribution function is 


B B 
Fs,.41|5, (t| $k) =i1-e” (« ~*k 


METHOD 2 (MAXIMUM LIKELIHOOD ESTIMATOR). Suppose we observed N 
failures at times S; = s1,...,5n = sn. We write the likelihood function as 


La, B| s1,...,8n) = fonisy_i ($n | 8N-1) fsy_i|Sw_o(8N—1|8N—2) °° * 
fs\s, (82 | $1) fs, ($1) 
= a8 sh le-* (oh 08-3) ; aB se e* (eh -s) es Aveee oe 


ap ee (s8—-s9) é aB Ce 3? 


N 
= (as)™ (TT sx)? ees, 
k=1 


Next, we find the expression for the log-likelihood function. We have 


2 


In(L) = Nina+Nin8+ (6-1) S- In(s,) — ash. 
k=1 
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Differentiating the log-likelihood function with respect to a and £ and setting 
the expressions equal to 0, we obtain 


Oa Sy 
and 
din(L) N WS 
—— = B + », In(sz) — ask, In(sy) = 0. 


From the first equation, ase, = N, and thus, the second equation can be 
rewritten as 


N 
+ S- In(s,) = ask, In(sv) = N In(sy). 
k=1 


From here, 


N 7 N 4 7 : 
NIn(sw) — 2), In(sx) = jm) ye In(sw/sx)] ene = IN Bie: 


Going back to our numeric example, we write syntax in R, utilizing the vari- 
able x that contains the logs of failure times. 


B= 


print (beta2.hat<- N/(N*x[N]-sum(x))) 
1.236357 

print (alpha2.hat<- N/exp(x[N]*beta2.hat) ) 
0.4863609 


(b) Another essential question in reliability analysis concerns the prediction 
of the next failure time. Suppose we have observed N failure times and the last 
one was at sy. We can estimate S41 by its conditional mean value. We write 


B(Swai Sw = sw) = [ cB 18 e-2 (t? sh) dt. 


SN 


Now we use the substitution u = at’, from where aft'’dt = tdu = 
a7!/8 y1/8 du. So, we obtain 


1(Sn 11| Sn = sy) = q7/B etsy / ui! e—” du. 


N 
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The integral can be calculated in R as an upper incomplete gamma function 


J-° u2-+e-“ du with the parameter a = 1/8 +1, and the lower limit of in- 


tegration t =a sh. The following code computes the prediction. It uses the 
maximum likelihood estimators of a@ and £. 


library (pracma) 

alpha2.hat’(-1/beta2.hat) *exp(alpha2.hat*exp(x([N] ) 
‘beta2.hat) * 

gammainc(alpha2.hat*exp(x[N])beta2.hat, 1/beta2.hat+1) [2] 


28.80161 


Thus, given that the 30th failure was observed at 28.05 weeks, we predict that 
the 31st failure will occur at 28.8 weeks. 


a 


Exercises 


EXERCISE 4.1. While still under the manufacturer’s warranty, calculators 
break down during the first three years with the rate of 3 per year. Between 
three and ten years, the rate increases linearly from 3 per year to 17 per year. 
(a) What stochastic model can be used to model the number of broken cal- 
culators? Specify all parameters. 

(b) Find the probability that 50 calculators break down between year 4 and 
year 8. 

(c) Find the average number of calculators that will break down between year 
2 and year 10. 


EXERCISE 4.2. Occurrence of wildfires in a certain area during a 120-day fire 
season can be modeled as a nonhomogeneous Poisson process with the inten- 
sity function A(t) = —0.000025 ¢? + 0.002 ¢? + 0.12t, where 0 < t < 120. 

(a) Plot the intensity function. Discuss its behavior. When is the peak of the 
intensity rate? 

(b) Plot the integrated intensity function. Find the average number of wild- 
fires per season. 

(c) Find the average number of wildfires during the middle 50% of the season. 


EXERCISE 4.3. Workers’ injuries at an industrial manufacturing plant oc- 
cur according to a nonhomogeneous Poisson process with the rate function 


A(t) = A/Vt, t > 0. 
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(a) Given that 30 injuries happened, on average, during the first year of 
plant’s operation, find the value of A. 

(b) Find the distribution of the times elapsing between two injuries. Simulate 
a trajectory of 100 injuries. What is the time range of the trajectory? 

(c) Assuming that the 100th injury occurred 12 years and 3 months after the 
plant was opened, simulate a trajectory. 


EXERCISE 4.4. Road traffic or airport noise data are often modeled over time 
as a nonhomogeneous Poisson process. It is assumed that noise pollution above 
a certain threshold has an intensity rate that allows cyclic behavior of observa- 
tions. For instance, suppose an environmental noise pollution has the intensity 
rate A(t) = 10 x (1 + cos(2zt)), t > 0. Use the thinning method to simulate a 
trajectory on the interval of length 10. 


EXERCISE 4.5. In the process of radioactive decay, photons are emitted ac- 
cording to a nonhomogeneous Poisson process with the intensity rate A(t) = 
100e~°->*, t > 0. Use each of the three simulation methods to generate a tra- 
jectory. Fix parameters as 20 events for the first method, and the length of 
the time interval as 0.25 in the other two methods. 


EXERCISE 4.6. The National Weather Service website contains the data on 
fatal lightning strikes in the United States. The file https://www.weather.gouv/ 
media/hazstat/80years.pdf gives the counts of yearly lightning fatalities be- 
tween 1940 and 2019. 

(a) Plot the counts against year. Argue that the intensity rate function decays 
exponentially. Provide a possible explanation for this decay. 

(b) Details of US lightning deaths (in particular, the dates) are provided on 
the same website hitps://www.weather.gov/safety/lightning-victims. The data 
are given for 2006-2019. Use the data to support the statement that this natu- 
ral phenomenon is not governed by a nonhomogeneous Poisson process. Hint: 
Are the interarrival times exponentially distributed? 


EXERCISE 4.7. The capacity of cargo container ships and port terminals are 
traditionally measured in twenty-foot equivalent units (TEUs), the number of 
20-foot-long containers. Suppose that containership arrival to a port can be 
modeled by a nonhomogeneous Poisson process with the power-law intensity 
function \(t) = a8 t?-!, a,B > 0, t > 0. The data for 27 arrivals (in units 
of 10,000 TEUs) are provided in the table below. 
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; 1 : 1 : 


1 
2 
3 
4 
5 
6 
7 
8 
9 


(a) Estimate the parameters of the model using the regression approach. 

(b) Estimate the parameters of the model using the maximum likelihood ap- 
proach. 

(c) Predict when the next 10,000 TEUs arrive at the port. Use both estima- 
tors from parts (a) and (b). 


Oo 


Compound Poisson Process 


5.1 Definition of Compound Poisson Process 


A stochastic process {X(t), t > 0} is called a compound Poisson process! if 
N(t) 


X(t) = S- Y; where {N(t), t > 0} is a Poisson process with rate \, and Y;’s 


i=1 
are independent and identically distributed random variables which are also 
independent of {N(t), t > 0}. 


PROPOSITION 5.1. The mean and variance of a compound Poisson process are 
R(X (t)) = AtE(Y1) and Var(X(t)) = AtE(Y7). 


PRooF: The mean can be computed by conditioning on the value of N(t). 
We write ; N(t) 
(X(t) =E[E(X()|N@)] =E[E( > VilN@)] 
i=1 
= E[N()E(%)| = E(N(@)) E(MY1) = AtE(Y1). 


Likewise, we compute the variance by conditioning on the value of N(t). We 


get Var(X(t)) = E[Var(X (t) | N(t))] + Var[E(X(£)|N(E))] 
N(t) N(t) “ 
= s[Var( 5° ¥i | N(t))] + Var|E( ¥i|N())| = E(N(t)Var(¥i)) 


+Var(N(t)E(Yi)) = E(N(t))Var(¥1) + Var(N(t))(E(¥1))? = AtVar(¥;) 
+A#(E(¥i))” = At(E(¥7) — (E(4))”) + At (E(M1))? = ALE(Y?). 


1The first treatment of the compound Poisson process is attributed to Filip Lundberg, 
a pioneer of the actuarial collective risk theory. In 1903, he wrote his Ph.D. dissertation at 
the University of Uppsala titled “Approximations of the Probability Function/Reinsurance 
of Collective Risks” (in Swedish). 
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EXAMPLE 5.1. Visitors walk into a casino in Las Vegas according to a Poisson 
process with a rate of 50 per hour. Ten percent of them will not gamble at all, 
others will lose independently a random number of dollars which we assume 
has a Uniform(0, $1,500) distribution. We need to model the casino’s gain. 


To this end, we focus only on the gamblers who are entering the casino. 
Their arrival can be modeled by a Poisson process {N(t), t > 0} with rate 
(0.9)(50) = 45 per hour. Let Y; denote the amount each gambler loses. We 
are given that Y;’s are independent and uniformly distributed with mean 
$1,500/2 = $750 and variance ($1, 500)2/12. Consider X(t) = 1% Y;,, the 
total sum of money that the gamblers lose at the casino within t hours. It is 
driven by a compound Poisson process. 


(a) The casino’s expected gain during a 12-hour period can be computed as 
U(X (12)) = (45)(12)($750) = $405, 000.00. 


(b) The standard deviation of the gain is 


Var(X(12)) = 4/(45)(12)(($1, 500)2/12 + ($750)?) = $20, 124.61. 


EXAMPLE 5.2. Suppose the number of car accidents at a certain intersection 
can be modeled by a Poisson process with rate \ = 3 per month. Assume that 
the number of people who are involved in each accident is a binomially dis- 
tributed random variable with parameters n = 8 and p = 0.3. Then the total 
number of people involved in car accidents on that intersection within a time 
period of ¢ months can be modeled by a compound Poisson process {X (t) = 
ye Y;, t > 0} where N(t) ~ Poisson(3t), and Y;, i = 1,2,...,N(t), are 
independent random variables with Bi(8,0.3) distribution, also independent 
of N(t). The average number of people involved in car accidents on this in- 
tersection within one year is E(X(12)) = At np = (3)(12)(8)(0.3) = 86.4 with 
the standard deviation 


Var(X (12)) = \/AtE(¥2) = \/At (Var(¥1) + (E(%))2) 


= Jt (np(1 — p) + n2p?) = ,/(3)(12)((8)(0.3)(0.7) + (8)2(0.3)2) = 16.36582. 


EXAMPLE 5.3. Families enter a movie theater according to a Poisson process 
with rate \ = 15 per hour. The number of family members is distributed 
according to a zero-truncated Poisson distribution with rate 4 = 3. We are 
interested in modeling the total number of moviegoers who enter the movie 
theater by time t. 
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Denote by {N(t), t > 0} the Poisson process that describes family arrivals, 
and let Y;, i = 1,2,..., N(#), be the size of the ith family entering. We are 
given that N(t) ~ Poisson(15t) and is independent of Y;’s, which, in turn, 
are independent and identically distributed with the probability mass function 
A oe 
PY, = 1) aired 1—e-’ 
be described by a compound Poisson process {X(t) = yO Y;, t > 0}. To 
find the mean and variance of this process, we first derive the expressions for 
the mean and second moment of a zero-truncated Poisson distribution. We 
write 


n=1,2,.... The total number of movie goers can 


Thus, the average number of people who enter the movie theater during a 
t-hour period is 


(X(t) = AtE(Y1) = =—— 


with the standard deviation 


Var(X(b) = 1/ MtE(Y2) = [Abas ve 


For instance, during a 6-hour period, the movie theater can expect E(X(6)) = 


1 3 
BVO = 284.1468 visitors, with a standard deviation of \/ Var(X (6)) = 
e3 — 


(15)(6)(3)(3 + 1) e8 
ee —1 


= 33.71331 visitors. 


5.2 Simulations in R 


In this section, we use the setting of Example 5.1 and simulate trajectories 
of the casino’s gain. First, we generate a Poisson process of gambler’s arrivals 
and then generate the amounts lost by the gamblers, which are independent 
random variables uniformly distributed on (0,$1,500). The sum of the loss 
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amounts up to time t is the desired compound Poisson process. We generate 
the Poisson arrivals by the two methods described in Chapter 3, by fixing the 
number of arrivals at 20 gamblers and generating independent exponentially 
distributed interarrival times, and by fixing the time interval at 20 minutes 
and generating event times as the order statistics from the uniform distribu- 
tion on [0,20]. The codes and graphs are given below. 


SIMULATION 5.1. (EXPONENTIAL INTERARRIVALS). 


#specifying parameters 
lambda<- 0.75 
narrivals<- 20 


#defining casino gain and time as vectors gain<- c() 
time<- c() 


#setting initial values 
gain[1]<- 0 
time[1]<- 0 


#specifying seed 
set .seed (50094) 


#simulating trajectory 

for (i in 2:(narrivalst+1)) { 

time[i]<- time[i-1] - 1/lambda*log(runif (1)) 
gain[i]<- gain[i-1] + runif(1,0, 1500) 

} 


#plotting trajectory 
plot(time, gain, type="n", ylim=c(0,14000), xlab="Time 
(min)", ylab="Casino gain ($)", panel.first = grid()) 


segments (time[-length(time)], gain[-length(time)], 
time[-1]-0.15, gain[-length(time)], lwd=2, col="blue") 


points(time, gain, pch=20, col="blue")) 
points(time[-1], gain[-length(time)], pch=1, col="blue")) 
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time [length (time) ] 
22.561 

gain[length (gain) ] 
13291.65 


In this simulated trajectory, 20 gamblers walked into the casino (a pre- 
determined number of arrivals). They all came within 22.561 minutes and 
lost cumulatively $13,291.65. 


SIMULATION 5.2. (UNIFORM ORDER STATISTICS). 


#specifying parameters 
t<- 20 
lambda<- 0.75 


#specifying seed 
set .seed (41130) 


#generating number of arrivals 
narrivals<- rpois(1,lambda*t) 


#defining vectors 
gain<- c() 

time<- c() 

u<- c() 
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#setting initial values 
gain[1]<- 0 
u[1]<- 0 


#generating standard uniforms and gain 
for(i in 2:(marrivalst+1)) { 

uli]<- runif (1) 

gain[i]<- gain[i-1] + runif(1,0, 1500) 


a: 


#computing event times 
time<- t*sort(u) 


#plotting trajectory 
plot(time, gain, type="n", ylim=c(0,13000), xlab="Time 


(min)", 


ylab="Casino gain ($)", panel.first = grid()) 


segments (time[-length(time)], gain[-length(time)], 
time[-1]-0.15, gain[-length(time)], lwd=2, col="blue") 


points(time, gain, pch=20, col="blue") 
points(time[-1], gain[-length(time)], pch=1, col="blue") 


Casino gain ($) 
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narrivals 

16 
gain[length (gain) ] 

12504.86 


In this simulated trajectory, within a fixed time period of 20 minutes, 16 gam- 
blers walked into the casino and lost a total of $12,504.86. 


5.3 Applications of Compound Poisson Process 


APPLICATION 5.1. A compound Poisson collective risk model is a classical 

model in the actuarial field. It assumes that claims are submitted according 

to a Poisson distribution with rate A, and that the amount of claims have 

a certain known distribution. Then the aggregate claim amount up to time t 
N(t) 

is a compound Poisson process { X(t) = x Y;, t > 0} where N(t) denotes 
i=1 

the number (or frequency) of claims, and Y; is the amount (or severity) of the 

ith claim. As in any compound Poisson model, N(t) and all Y;’s are assumed 

independent. We know that E(X(t)) = \tE(¥1) and Var(X(t)) = AtE(Y?). 


The main question that actuaries have to answer is how much money should 
be collected in premiums so that the company will be able to pay the claims. 
Let L(t) = X(t) —ct be the insurer’s loss. It represents the difference between 
the total benefit payments that the company has to make and the amount 
of premiums collected over time with a constant rate c. It is customary to 
consider c of the form c = (1+ 0)AE(Y,) where @ is termed a security load- 
ing. It means that up to time ¢, the company will collect in premiums the 
amount ct = (1+ 0)AtE(¥1) = (14+ 6) E(X(t)), which gives the company 
some cushion above the expected aggregate claim amount E(X(¢)) in case 
there are some unusually high claims. One of several ways to find the value 
of the security loading @ is to assume that the company wants to see a posi- 
tive loss at most, say, 5% of the time. Thus, 0 solves P(L(t) > 0) = 0.05 or 
P(X(t) — (1+ 0) E(X(t)) > 0) = 0.05. We can rewrite this identity as 
p(XWaEXO) , 4 #80) 


> os) — 0.05. 
Var(X(t)) XEE(Y 2) 
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Now, assuming that A is large, we can use the Central Limit Theorem to con- 
X(t) — E(X(¢)) 
Var(X (t)) 


Hence, @ can be found as the solution of the equation 


P(Z > ss = 005, 


E(Y/") 


clude that has approximately a standard normal distribution. 


That is, 


E(Y;") 

V\tE(Y;) 

As an illustration, we consider data on storms downloaded from the National 
Oceanic and Atmospheric Administration’s site (https://www.ncdc.noaa.gou/ 
stormevents/). The data set contains dates, times, and amounts of damage (in 
units of $1,000) in all counties in Texas from March to April of 2020. The re- 
ported damage was done during a storm by hail, wind, tornado, flash flood, or 
lightning. There are a total of 85 rows in this data set. Damages range between 
$500 and $150,000, with two additional values of $500,000 and $800,000. As- 
suming that a single insurance company took care of all the claims, below we 
evaluate the security loading that this company must utilize when calculating 
premiums. 


6 = 1.645 


storm.data<- read.csv(file="./stormdata.csv", header=TRUE, 
sep=",") 


#creating date-time variable 
storm.data$datetime<- as.POSIXct 
(paste(as.Date(storm.data$Date), storm.data$Time) ) 


#estimating event rate 
print (nevents<- nrow(storm.data) ) 


85 


print (ndays<- (as.numeric(storm.data$datetime[nevents] ) 
-as.numeric(storm.data$datetime[1]))/(24*3600) ) 


46 .09028 
print (lambda. hat<- nevents/ndays) 
1.844207 


Within 46.09 days, there were 85 storms with tangible damage (resulting in 
the insurance claims). It means that the claims were submitted with a rate 
of 1.844207 per day. Finally, we estimate 0, using the empirical values of the 
first and second moments of the damage amounts. 


Applications of Compound Poisson Process 113 


#estimating security loading 
print (theta<- 1.645*sqrt (mean(storm.data$Damage’2) )/ 
(sqrt (lambda. hat*ndays) *mean(storm.data$Damage) ) ) 


0.5516492 


It means that the company should collect about 155% of the mean claim 
amount to hedge against large claims. 


TT 


Exercises 


EXERCISE 5.1. The producer of a television game show with cash prizes wants 
to set a budget equal to the expected value plus one standard deviation of the 
aggregate cash prizes. The number of cash prizes given is a Poisson process 
with a rate of 1.5 per hour. Each episode lasts for 2 hours. The distribution 
of prize amounts is as follows: 


$5,000 $2,000 5500 5100 
Probability [0.15 0.385 02 0.3 


(a) Calculate the budget for 100 episodes of the game show. 

(b) Simulate a trajectory of 100 games. For the trajectory, what is the amount 
of the budget left after the 100th game? If the amount is negative, during or 
after which episode did the producer run out of money? 


EXERCISE 5.2. When a pharmacy bills the medical insurance company, the 
claims arrive as a Poisson process with the rate \ = 60 per day. The amounts 
of claims are independent and uniformly distributed between $30 and $300. 
It is also assumed that the amounts of the claims and the number of claims 
are independent. 

(a) What is the expected aggregate claim amount that the medical insurance 
company receives within a 30-day period? What is the standard deviation of 
this amount? 

(b) Use the Central Limit Theorem to approximate the probability that the 
aggregate claim amount will exceed $300,000 within a 30-day period. 


EXERCISE 5.3. The photon detection process in X-ray computed tomography 
can be modeled as a compound Poisson process. The X-ray photons collide 
with a photon detector and then generate some number of light photons. The 
number of incident X-ray photons changes according to a Poisson process with 
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a rate of \ per second. The number of light photons generated by each X-ray 
photon that is detected is a Poisson random variable with a rate of per 
second. 


(a) Define the aggregate number of light photons that are generated up to t 
seconds. Give the formula for the process, and the expressions for its mean 
and standard deviation. 

(b) Assuming that the average rate of X-ray photons is 50 per second and 
the mean of light photons is 5, simulate 100 values of the aggregate number 
of light photons generated within a 10-second period. 

(c) Construct a histogram for the 100 values generated in part (b). Is the 
histogram approximately bell-shaped? Explain. 


EXERCISE 5.4. An insurance company receives claims according to a Poisson 
process {N(t), t > 0} with rate \. Assume that claim amounts X;, 7 = 1, 2,..., 
are independent and identically distributed, and are independent of claim 
arrival times S;,i = 1,2,.... The present-day (day of policy issue) value of 
the amount of claim X; made at time S; is computed as X; erie, where 
6 is the force of interest. The present-day value of the total claim amount 
P(t) = pena X;,e~°% changes according to a compound Poisson process. 
Show that the mean of this process is E[P(t)] = E(X1)(A/d)(1— 7°’). 


EXERCISE 5.5. In radiobiology, when cells are exposed to radiation, DNA 
sometimes breaks, and the broken ends may abnormally rejoin resulting in 
chromosome aberrations. The number of ions that traverse through a cell nu- 
cleus by time t is modeled as a Poisson process with rate \. Each traversal 
independently causes Y; aberrations which have Poisson distribution with rate 
B. Let X(t) = pant ) Y; denote the total number of chromosomal aberrations 
by time t. 

(a) Show that the compound Poisson process X(t) has the Neyman type A 
distribution with the probability function given by 


P(X()=2) = Fert ye PON on 
; n=0 . 


(b) Show that the mean of X(t) is \t@ and the variance is At(G + 87). 

(c) For a fixed t, denote the sample mean of X(t) by E(X(t)), sample vari- 

ance by Var(X(t)), and empirical probability of zero P(X(t) = 0) by P(0). 
Var(X(t)) 


Show that 6 and may be estimated respectively by B = ———— — land 
E(X(t)) 
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EXERCISE 5.6. Cars arrive at a gas station according to a Poisson process. 
Each car driver buys gas independently of others for a dollar amount that 
has a gamma distribution. The spent amounts and the number of cars are 
independent. Data for times of car arrivals (in minutes) and dollar amounts 
spent are presented in the table below. 


Arrival Amount | Arrival Amount | Arrival Amount 
Time Spent,in$] Time Spent, in $ | Time Spent, in $ 
5 : 28.81 : 47.94 ; 


(a) Argue that the total dollar amount that the gas station receives can be 
modeled by a compound Poisson process. Write down the expression for the 
process and describe all parameters. 

(b) Estimate the parameters of the model using the method of moments. 
Show that the estimators of a and ( of the gamma distribution are @ = 


2 
aan and B => a 
ee = 
(c) Plot histograms for the interarrival times and the amount spent, with fit- 
ted distribution curves. 
(d) Write down the estimates of the mean and standard deviation of the to- 


tal dollar amount at 1 hour. Use the parameter estimates obtained in part (b). 
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Conditional Poisson Process 


6.1 Definition of Conditional Poisson Process 


A counting process {N(t), t > 0} is called a conditional (or mixed) Poisson 
process! if N(t) has a Poisson distribution with rate A where A is a random 
variable with a known distribution f,(A). The rate A is referred to as a ran- 
dom intensity rate. 


The probability mass function for the conditional Poisson process is specified 
as a conditional probability 


Ce 


a ge > 0) mS, as 
ne 


P(N(t+s)—N(s) =n|A=A) = 


The marginal distribution of N(¢) is found as 


a FO) Oy b> 06 Lae. « 


P(N(t+s) — N(s) =n) -[ 


REMARK 6.1. A conditional Poisson process has stationary increments. It is 
reflected in the above formulas for conditional and marginal distributions of 
N(t). The above expressions do not depend on s, the beginning of the inter- 
val, only on t, the length of the interval. The increments are not independent, 
though. It is easily seen if we write 


P(N(s) =n, N(t—s) =m) = [ Pine) =n, N(t—s) =m|A=X) fa(A) dr 


-[ P(N(s) =n|A=A) P(N(t— s) =m|A=A) fa(A) dr 


lntroduced in Dubourdieu, J. (1938). “Remarques relatives a la théorie mathématique 
de l’assurance-accidents.” Bulletin Trimestriel de l’Institut des Actuaires Francais, 49: 76 
— 126. 
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an P(N(s) =n|A=A) fa(A) da i P(N(t—s)=m|A=A) fa(A) da 
0 0 


= P(N(s) =n) P(N(t— s) =m). 


PROPOSITION 6.1. The mean of N(t) is E(N(t)) = tE(A), and the variance 
is Var(N(t)) =t? Var(A) + tE(A). 


PROOF: Conditioning on A, we write 


E(N(t)) = E[E(N(t)|A)] = E(At) = #E(A), 


and 


Var(N(t)) = Var[E(N(t) | A)] + E[Var(N(é) | A)] 
= Var[At] + E[At] = 2? Var(A) + tE(A). 


EXAMPLE 6.1. Assume that A in a conditional Poisson process has the prob- 
ability mass function 


P(A — Xo) = Po and P(A _ M1) =] — po. 
The marginal distribution of N(¢) is 


(Ao t)” enrol y (Ai t)” 


= = Arty = 
P(N(t) =n) = Al o + ne (1 —po),n =0,1,2,.... 


This is a mixture of two Poisson processes with rates Ay and A; and the mix- 
ture weights po and 1 — po. As a mixture of two Poisson processes, N(t) has 
mean E(N(t)) = Aotpo + Art (1 — po) = t (Ao Po + A1(1 — po)) = tE(A), and 
variance 


Var(N(t)) = po(Ao t+ (Ao t)?)+(1—po)(A1 tt+(a1 t)?)—(Xo tpo + Art (1—po))? 
= t? (Np po + AZ (1 — po) — (Ao Po + A1 (1 — po))”) + t (Ao Po + Ai (1 — Po)) 
= #2 Var(A) +tE(A). 


EXAMPLE 6.2. Suppose that in a conditional Poisson process, A has an expo- 
nential distribution with mean \. The marginal probability mass function of 
N(t) is 


=n)= ae (ut)” enw i eT Ula a 
P(N(t) = =f d 


f° eet en ttl) ay 
= GUN yr+t 

_ A Cl ( 1 
(t+ 1/dAy"*41d)  At+1 At+1 


) SOCIO 3, 
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Therefore, the marginal distribution of N(t) is geometric. It represents the 
number of failures until the first success. The success probability is p = 1/(At+ 


1- 1-1/(At+1 
1). It is known that the mean is es pe Cae) = At =tE(A), and 
1/(At+1) 


; a ge Lae). _ 4252 2 
variance is 2 =Toteie M+ tA =t Var(A) + 
tE(A). 


ee 
6.2 Simulations in R 


SIMULATION 6.1. (EXPONENTIAL INTERARRIVALS). Consider the setting of 
Example 6.1. Suppose pp = 0.3,A9 = 4, and A; = 0.5. Below we simulate 
5 trajectories with 20 Poisson occurrences each, by generating exponential 
interarrival times. We first choose the rate randomly from the given binary 
distribution. Three of the trajectories have a rate of 4 and the other two have 
a rate of 0.5. 


#specifying parameters 
p<- c(0.3, 0.7) 
lambda<- c(4, 0.5) 
njumps<- 20 


#specifying states and times as data frames 
time<- data.frame() 
N<- data. frame() 


#specifying seed 
set .seed (6335044) 


#simulating trajectories 
for(j in 1:5) { 


#fixing the value for rate 
Lambda<- lambda[sample(1:2, 1, prob=p)] 


#setting initial values 
time[1,j]<- 0 
N(1,j]<- 0 


#simulating trajectory 
i<- 2 
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repeat { 

time [i,j]<- time[i-1, j]+round((-1/Lambda) *log(1-runif (1)) ,3) 
-0.001 

N[i,j]<- N[i-1,j] 


if (i==2*njumps+2) break 
else { 
time [it+1,j]<- time[i,j]+0.001 
N[it1,j]<- N[i,j]+1 
T<> a2 


u 


#plotting trajectories 

matplot(time, N, type="1", lty=1, lwd=2, col=c("blue", 
"green", "red", "purple", "orange"), xlab="Time", 
ylab="State", panel.first=grid(Q) 


20 5 


15 5 


10 5 


State 


SIMULATION 6.2. (UNIFORM ORDER STATISTICS). Now we use the uniform 
order statistics method described in earlier chapters to simulate five trajec- 
tories. We fix the end-time (say, t = 30), and then pick the rate \ from the 
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binary distribution. After that, we randomly choose the number of events from 
the Poisson (At) distribution. Note that we expect trajectories with A = 0.5 
to have about (30)(0.5) = 15 occurrences, whereas trajectories with A = 4 
will have about (30)(4) = 120 occurrences. Below is the code and the graph. 
Two trajectories happened to have a rate of 4 and the other three have a rate 
of 0.5. 


#specifying parameters 
t<- 30 

p<- c(0.3, 0.7) 
lambda<- c(4, 0.5) 


#specifying states and times as data frames 
time<- data.frame() 
N<- data.frame() 


#specifying seed 
set .seed (1902238) 


#simulating trajectories 
for(j in 1:5) f 


#fixing the value for rate 
Lambda<- lambda[sample(1:2, 1, prob=p)] 


#setting initial values 
time[1,j]<- 0 
N(1,j]<- 0 


#generating total number of jumps 
njumps<- rpois(1,Lambda*t) 


#generating standard uniforms 
<= ct) 

uli]<- 0 

for(i in 2:njumps) 

uli]<- runif (1) 


#computing event times 
s<- t*sort(u) 
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for (i in seq(2, 2*njumps, 2)) { 
time [it1,j]<- s[i/2] 


time[i,j]<- s[i/2]-0.001 


#generating jumps 


NCi jl Ss N(i-1 jl 


N({it1,j]<- N[i-1,j]+1 


} 


#plotting trajectories 


a: 


c("blue", 


1, lwd=2, col= 
"Time" , 


Be Be ese eee Ser ae 


grid()) 


' 
' 
1 

7 
' 
1 
1 
1 
1 
1 
' 
1 
1 
' 
1 

1 
1 
' 
' 
' 
1 


see se esmemne So coeneeee 


matplot(time, N, type="1", lty 
"green", "red", "purple", "orange"), xlab 


ylab="State", panel.first 


100 +-- 


BOi<1=- 
60 +-- 


BYES 


bee Sen.5.5.S55 


40 +-- 
0d = 
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6.3 Applications of Conditional Poisson Process 


APPLICATION 6.1. Agronomists model the presence of Colorado potato bee- 
tles as a conditional Poisson process. Suppose a plot of fertile soil is planted 
with potatoes. The width of the plot is fixed, so the area is proportional to the 
length. The number of egg clusters A that are located within a stretch of the 
plot of a certain length has a Poisson distribution with mean Ag. Each cluster 
contains between 10 and 30 eggs, but only several of them hatch. Suppose the 
number of eggs within a cluster that hatch follows a Poisson distribution with 
a rate r. Combining the results, we obtain that the total number of hatched 
eggs N(£) on a stretch of the plot of length @ has a Poisson distribution with 
the rate r A, where A ~ Poisson(Xo). 


(a) The expected value of N(£) is computed by conditioning on A. We 
write E(N(£)) = E[E(N(€)|A)] = E(r@A) = rAoé. Likewise, the variance 
is computed by conditioning on A. We have Var(N(é)) = Var[E(N(2) | A)] + 

=[Var(N (2) |.A)] = Var(r@A) + E(r€A) = (ré)?X0 + (ré)Ao = Ao(ré)(ré + 1). 


To illustrate these with a numeric example, assume r = 5,f = 2, and Ag = 3. 
We evaluate E(V(@)) = (5)(3)(2) = 30 and Var(N(€)) = (3)(5)(2)((5)(2) + 
1) = 330. 


(b) The marginal distribution of N(¢) is 


P(N(é) =n) = E[P(N(é) =n|A)] = gf PAT one 


n! 


e)" Lyn 

mi uA" e | = (r)" Me) (re), 01.0 ,, 
n! n! 

where (7) (re) is the nth derivative of the moment generating function of 

A, Ma(t) = exp{o (e’ — 1)}, computed at t = —ré. These derivatives have 

to be computed numerically. Below is the R code that calculates probabilities 

forn =0,...,10. 


#specifying parameters 


r<- 5 
<2 
t<- -r*1 


lambda0<- 3 


124 Conditional Poisson Process 


#computing probabilities 

prob<- c() 

M<- expression (exp (lambda0*(exp(t)-1))) 
prob[1]<- eval (M) 


for (m in 2:11) { 


M<- D(M,"t") 

prob[n]<- (r*1)n/factorial (n) *eval (M) 
} 

prob 


[1] 0.0497938498 0.0003390956 0.0011304726 
[4] 0.0028269513 0.0056569821 0.0094385700 
[7] 0.0135130114 0.0169646392 0.0190127388 
[10] 0.0193392713 0.0181755410 


APPLICATION 6.2. An auto insurance company models the number of claims 
up to time t, N(t), as a Poisson process with random intensity rate A which 
represents an accident-proneness index of a policyholder. The values of A are 
distributed according to a gamma distribution with mean a/3 and variance 


of. 


(a) The expectation and variance of N(t) are E(N(t)) = tE(A) = at/f, and 
Var(N(t)) = t?Var(A) + tE(A) = at?/B? + at/B. 


To compute these quantities for some specific values of the parameters, let’s 
assume that a = 0.3 and 8 = 1. The average total number of claims per 
policyholder that the company has to deal with every 5 years is E(N(5)) = 
(0.3)(5)/(1) = 1.5. The standard deviation is 


VVar(N(@)) = V(0.3)(5)2/C? + (0.3)(6)/) = 3. 


(b) The marginal probability mass function of N(t) has the form 


Piv(t) =n) = fo SR XE 


oe T'(n+a) Bo - Arr a Bre ett B)A gy 
nm! T(a) (¢+6)rte Jo T'(n + a) 


aguas 
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Therefore, N(t) has a gamma-Poisson mizture distribution, which has the 
algebraic form of a negative binomial distribution with parameters a and 


p= nop Since a@ assumes real values that are not restricted to integers, this 
is not a true negative binomial distribution. The mean and variance of this 
distribution are 


_ a(l-p) _ at/(t+8) 


o(N(t)) = ap EB, = at/B, 


and 


Var(N(t)) = a a eet = a“ =at?/B? + at/B. 


These coincide with the expressions derived in part (a). 


(c) We can also compute the conditional probability that A, the accident- 
proneness index of a policyholder, doesn’t exceed some particular value 4, 
provided that the policyholder has made n claims within t years. We derive 


A (ut)” ~—ut u®71 8% (-Bu 
P(A <A|N(t)=n) = PA Man ASN) Jo Sat eo ah oP 
= > = = ~ oo (ut) Lye Peo Be By 
P(N t) =n) fe ui)” e rae B du 


Xr nra— wae i ae ne 
utente" du ie EE BIPM Cet dy 
0 


— i yrta-1 e—(t+B8)u dy = T(n + Q) 


This is a gamma distribution with parameters n + a and t+ (. The expected 
value of A for a policyholder who is known to have made n claims within t 


years is E(A| N(t) =n) = te 


For a = 0.3 and 8 = 1, the expected accident-proneness index for a poli- 
cyholder who has made two claims in 5 years is E(A|.N(5) = 2) = = = 
0.3833. 


a 


Exercises 


EXERCISE 6.1. Let {N(t), t > 0} be a conditional Poisson process with the 
random intensity rate A. Show that for any t > s > 0, 

(a) Cou(N(s), N(t) — N(s)) =s(t—s) Var(A). 

(b) Cov(N(s), N(t)) = st Var(A) + sE(A). 
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EXERCISE 6.2. Suppose {N(t), ¢ > 0} is a conditional Poisson process and 
the random intensity rate A has density f,(A), \ > 0. Show that 
(a) The conditional cumulative distribution function of A, given N(t) =n, is 


_ fc u®e—%® Fx (u) du 


ee ur et fa(u) du 


Faw) (Ain) = P(A < A| N(t) =n) 
(b) The conditional density function of A, given N(t) = n, is 


rn ert faa) 
Xr ae a ee 
faiway (Aln) Te Pe fan 
(c) The conditional expected value of A, given N(t) = n, is 
_ Sg Atte falar 
~ fp eax 


[A | N(t) =n] 


EXERCISE 6.3. Suppose that 46% of all visitors of an amusement park are 
teenagers, 24% are adults, and the rest are kids. Assume that the number 
of visitors who come into the park during a busy hour can be modeled as a 
Poisson process with the random intensity rate that varies depending on age 
group: 4 per minute for teens, 2 per minute for adults, and 3 per minute for 
kids. 

(a) Write down the model and specify all parameters. Find the mean and 
variance of the number of visitors within time t. 

(b) Simulate five trajectories of the process with 200 visitors each. 

(c) Simulate five trajectories of the process that depict arrivals within 1 hour. 


EXERCISE 6.4. A credit union assigns to all its clients a rating value A in such 
a way that a client defaults on a credit account according to a Poisson process 
with rate A (per year). The distribution of A is uniform on [0, 2]. 

(a) Find the average number of defaults that a client has within a 5-year pe- 
riod. 

(b) Find the variance of the number of defaults within a 5-year period. 

(c) Find the covariance between the number of defaults during the first 3 
years and that during the subsequent 2 years. Hint: See Exercise 6.1(a). 

(d) Find the covariance between the number of defaults during the first 3 
years and that during the first 5 years. Hint: See Exercise 6.1(b). 

(e) Find the probability that the client’s rating value is less than 0.5, given 
that he has had two defaults within a 5-year period. Hint: See Exercise 6.2. 


EXERCISE 6.5. Snow Water Equivalent (SWE) describes the amount of water 
contained within a snowpack, measured in inches. It is an important notion 
in environmental science, agriculture, and forestry. An annual SWE is well 
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modeled by a conditional Poisson process. The amount of SWE is a Poisson 
process with rate A inches per year that is itself a Poisson random variable 
with a rate of 24.3. 

(a) Compute the average and standard deviation of SWE for a 1-year period. 
For a 5-year period. 

(b) Simulate five trajectories that reach 140 inches of SWE each. 

(c) Simulate five trajectories spanning over a 7-year period. 


EXERCISE 6.6. In the textile industry, a series issue is the number of defects in 
fabric per linear footage. Suppose quality control engineers model the number 
of defects as a conditional Poisson process with a random intensity rate A per 
yard for the fabric of standard width. The random variable A has a gamma 
distribution with a mean of 0.07 and a standard deviation of 0.01. 

(a) Compute the expected number of defects in a 40-yard roll of fabric. Find 
the standard deviation of the number of defects in the roll. 

(b) Given that four defects were found in a 40-yard roll of fabric, find the 
probability that A exceeds 0.08. 
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Birth-and-Death Process 


7.1 Definition of Birth-and-Death Process 


A continuous-time Markov chain {X(t), t > 0} with state space S = 
{0,1,2,...} is called a birth-and-death process! if, given that the chain is 
in state n, the time to transition to state n + 1 is exponentially distributed 
with mean 1/A,,, and the time to transition to state n — 1 is exponentially 
distributed with mean 1/,. The two waiting times are independent. 


PROPOSITION 7.1. Ina binohe and-death pee the transition probabilities 
are Po91 =1, Prnsti = x An , and Prn-1 = os care . All the other transition 
prabatiaes are 0’s. The one-step transition probability matrix has the form 


0 1 2 3 
0 0 1 0 0 
a 
1 rau : ve aT : 
Pe 2 0 2 + ii 0 2 mG 
3 | 0 0 ws 9 


PROOF: Consider two independent exponential random variables Tg and Tp 
with means 1/A and 1/, respectively. The variable Tg represents the wait- 
ing time until a “birth,” and Tp represents the waiting time until a “death.” 
A “birth” occurs before a “death,” if Tg < Tp. The probability of this event is 


P(Tp < Tp) =| - Oe eM dyde =f Ne A* eH dx = ——_. 
A+ pb 


1The first example of a birth-and-death process was described in 1939 by William Feller, 
a renown Croatian-American mathematician, in “Die Grundlagen der Volterraschen Theorie 
des Kampfes ums Dasein in wahrscheinlichkeitstheoretischer Behandlung.” Acta Biotheo- 
retica, 5: 11 — 40. 
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Analogously, a “death” occurs before a “birth” if Tp < Tg, which happens 
with the complementary probability 


A _ 
At Atp 


P(Tp < Tg) =1-—P(Tp < Tp) =1- 


PROPOSITION 7.2. Suppose a birth-and-death process is in state n. Then the 
waiting time until a transition occurs is exponential with mean 1/(An + pn). 


PROOF: Referring to the proof of the previous proposition, we see that a 
transition occurs at time Tg or time Tp, whichever happens first. That is, we 
need to find the distribution of min(T'g, Tp). We write 


Fnin(Ts,Tp)(t) =P(Te < t, Te < Tp) + P(Ip < t, Tp < Tp) 


t foe) 
-[{f[ re ** pe “erdyae + ff Ne ** we“ dx dy 
0 vy 
t 
Sif Nees B® da +f e*¥ we #Y dy 
0 


-[ (At pe OTH du = 1 — e OTHE, 
0 


which is an exponential distribution with mean 1/(A + j). 


PROPOSITION 7.3. Consider a birth-and-death process, and denote by P,,(t) 
the probability that at time t the process is in state n. The probabilities 
P,(t), 2 =0,1,..., satisfy the system of the Kolmogorov forward equations: 


Py (t) = —Ao Po(t) + yu Pi(t), 
POS Raha) Hi Po) SO PO RED 
(7.1) 
with the boundary condition P,,,(0) = 1. 


PROOF: We will omit the rigorous derivation of these equations but will ex- 
plain their simple meaning. How can the process transition into state n? Only 
if there are n—1 particles and one more is born, or there are n+1 particles and 
one dies. That is, An—1 Pn—1(t) + Un+i Pr+i(t) represents the rate of change 
with respect to time of probability to transition into state n. By the same 
token, (An + fn) Pr(t) gives the rate of change over time of probability to 
transition out of state n (one particle is born or dies). Thus, the expression on 
the right is the difference between the in and out rates, and so is the meaning 
of the derivative P, (t) on the left. 
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REMARK 7.1. The mean and variance of {X(t), t > 0}, a birth-and-death 
process, are computed as 


E(X(t)) = S$" n Pa(t), and Var(X(t)) = S > n? P,(t) - [E(x (t))]”. 
n=0 n=0 


EXAMPLE 7.1. A Poisson process is an example of a birth-and-death process 
with A, = A and uy, = 0, for n = 0,1,.... Because there are no “deaths,” it is 
a pure birth process. 


(a) We know that in this process, times until “births” are independent and 
exponentially distributed with mean 1/\, = 1/2. 


(b) The transition probabilities are Po, = 1, Panta = An/(An + in) = 
A/(A +0) = 1, and Prin-1 = bn/(An + bn) = 0/(A+ 0) = 0, which we know 
is true since only jumps of size +1 are admissible in a Poisson process. 


(c) The probabilities P,,(t), = 0,1,..., satisfy the Kolmogorov forward 
equations P; (t) = —A Po(t) and P/ (t) = \ Pn_1(t) — A Pa(t), forn = 1,2,..., 
with the boundary condition P (0) = 1. The solution to these equations is the 
Poisson probability mass function P,(t) = ey et, n = 0,1,...,, which 
can be verified by writing 


P, (t) = (e~>*)' = —Ae** = —d Pols), 


f =. (At)” —rt\' - An" n—-1 n\ ,-At _ (Ato At 
POa (Se) Set Pare an ae 
(ADI ® 2 0° 9 
~STE eM = AP alt) — APa(t), and Fh(0) = Te? = 1. 
nT i 


(d) Both the mean and variance of the Poisson process, as we know, are equal 
to At. 


EXAMPLE 7.2. A linear birth-and-death process { X(t), t > 0} is a birth-and- 
death process with the parameters A,, = nA and py = wn, n=0,1,.... Note 
that this process state 0 is an absorbing state. We assume that the process 
starts in state 1. 
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(a) In this process, times until “births” and “deaths” are independent and 
exponentially distributed with mean 1/Ap, = 1/(nA) and 1/un = 1/(np), re- 
spectively. Time until a transition (“birth” or “death”) are independent expo- 
nentially distributed random variables with mean 1/(An + fn) = 1/[n(A+p)]. 


(b) The transition probabilities are Po, = 0, Panti = An/(An + in) = 
A/(A+ f), and Prn-1 = Mn/(An + Mn) = B/(AF pB)- 


(c) The Kolmogorov forward equations in this case assume the form: P; (t) = 
pu P;(t) and for n = 1,2,..., P(t) = (n—1)APy_i(t) + (n+ 1) Prsi(t) — 
n(X\ + 1) P,(t), with the initial condition P,(0) = 1. The solution of these 
equations can be written as 


and 


P,(t) = (1— Po) (1 — = P)(SP)" n=1,2,.... 


This distribution is a mixture of a point mass at zero and a geometric distribu- 
tion modeling the number of trials until the first success where the probability 


of success is p= 1 — * Po. 


(d) The mean of the process can be computed as 


i: i ps 
= — = 
Dok Tee Ep 


E(X(t)) = (1 — Po) 


The variance is equal to 


Var(X(t)) = (1 — Py) 7 = Soe eltane(etrmot diye 


a Ee 


7.2 Simulations in R 


SIMULATION 7.1. Below we simulate a 20-step trajectory of a linear birth- 
and-death process with parameters \ = 0.3 and yp = 0.1 that starts at time 0 
in state 1. We generate two independent exponential times with rates 1/(nA) 
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and 1/(ny), and transition 1 unit up if a “birth” occurs before “death” or 1 


unit down, otherwise. 


#specifying parameters 


lambda<- 0.3 
mu<- 0.1 
njumps<- 20 


#setting state and time as vectors 
N<- c() 
time<- c() 


#setting initial values 
N[1]<- 1 
time[1]<- 0 


#specifying seed 
set .seed(1022171) 


#simulating trajectory 
i<x- 2 


repeat { 
time.birth<- (-1/(N[i-1]*lambda) )*log(runif (1)) 
time.death<- (-1/(N[i-1] *mu) )*log(runif (1)) 


if (time.birth < time.death | N[i-1]==0) { 
time[i]<- time[i-1]+time.birth-0.001 
NCil<- N[i-1] 


if (i==2*njumps+2) break 
else { 
time [i+1]<- time[i]+0.001 
N[iti]<- N[{iJ+1 
i<- i+2 


Bs 
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if (time.death < time.birth & N[i-1]!=0) { 
time[i]<- time[i-1]+time.death-0.001 
N[i]<- N[i-1] 
if (i==24njumps+2) break 
else { 
time [it+1]<- time[i]+0.001 
N({iti]<- N[{ij-1 


i<- it2 
he 
t 
t 
#plotting trajectory 
xlab="Time", 


plot(time, N, type="1", lty=1, lwd=2, col=4, 
ylab="State", panel.first=grid(Q) 


State 


Time 
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7.3 Applications of Birth-and-Death Process 


APPLICATION 7.1. An M/M/1 queue is a birth-and-death process { X(t), t > 
0} with A, = A and pw, = p. We assume  < wy. In this process, customers 
join a queue (representing “births”) at independent exponential times with 
mean 1/, and leave the system (representing “deaths”) after going through 
the service, which time is exponentially distributed with mean 1/. All cus- 
tomers act independently. In the name of the process, the first “M” means 
that the customer arrival process is Markovian, the second “M” refers to the 
fact that the service time is Markovian, and the “1” stands for a single server. 
An example of M/M/1 process is the number of customers in a bank with 
a single bank teller: customers enter the bank and join the line to the bank 
teller, then when it is their turn, they get a service from the teller and leave 
the bank. Another example is the number of broken cars in a repair shop with 
a single repairman: broken cars are added to the line to get a service from the 
repairman, and leave the shop once they are repaired. 


(a) The Kolmogorov forward equations for M/M/1 process are 


, 


Py(t) = -A Pot) + w Pit), 
P,, (t) = A Pn-1(t) + LU Pr+i(t) _— (A + Lt) Pr(t), feos V2 ek 


with the boundary condition Pp(0) = 1. The solution to these equations exists 
but is rarely used in practice. Instead, the limiting (or steady-state) probabil- 
ities are computed. They are defined as limy4. Pn(t) = Pr, n > 0. To find 
the limiting probabilities, we pass to the limit in the Kolmogorov forward 
equations as t tends to infinity. Replacing the left-hand side by 0 (since the 
derivative of a constant is 0), we obtain 


0= —APo + pP,, 
0 = APh-1 + w Pag — (A + Bw) Pa, 2 =1,2,.... 


These can be rewritten in the form of what is known as the balance (or equi- 
librium) equations: 


uP, = APo, 
(A + Lt) Pr = Pr—1 + Lb Pasi, WH AG Qoakes 
The expression on the left represents the mean rate of leaving the state n, 


whereas the right-hand side gives the mean rate of entering state n. Thus, the 
balance equations equate the mean departure rate and the mean entrance rate. 


To solve the balance equations, we notice that wP, — APp = O and 
bPa4i —APh = Pn -— APn-1 = ... = wP,-AP = 0. Therefore, 
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P= A p, = (ei a (Ap, n = 0,1,.... Since all 
Lt L bo 
2 A\n-1 
the probabilities must add up to 1, we conclude that Py = s (-) = 


m 

1 - a and so P, = (1 - “) (-) , n= 0,1,2,.... This is a geometric 
[ 

distribution modeling the number of failures before the first success, where 

the success probability is p= 1— A/p. 


(b) The average number of customers in the system, in the long run, is 
computed as 


(c) We will show that the amount of time T a customer spends in the system 
is an exponential random variable with mean 1/(j— A). Indeed, suppose when 
the customer arrives, there are already n customers in the system. If n = 0, 
then T is the service time which is exponential with mean 1/p. If n > 0, the 
customer will have to wait for one customer to complete the service, and then 
for n additional customers (including himself) to go through the service. Us- 
ing the memoryless property of an exponential distribution, we conclude that 
the waiting time, in this case, is the sum of n + 1 independent exponential 
with mean 1/y random variables, which is a random variable having a gamma 
distribution with mean (n + 1)/. We write 


co n, ntl 
fr(t) =E[fr(t|n customers)] = » oop e Ht Pp, 
= : Rees prety Pa 
= ee (12) (A) = ener AE A yen 
n=0 , n=0 . 


which is an exponential distribution with mean 1/(j— X). From here, we can 
conclude that the probability that a customer spends more than time t in the 
system is P(T > t) = e~-E, 


(d) Consider an M/M/1 system in which arrivals occur with the rate A = 1 
per minute, and departures occur with the rate = 1.5 per minute. The 
steady-state probabilities are 


eerie omer 


In the long run, there will be, on average, = 2 customers in 


ahs to 
the system. The probability that in a long run, a customer will spend over 5 
minutes in the system is P(T > 5) = e~“5-D) — 0.082085. 
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Exercises 


EXERCISE 7.1. A Yule process” (or a linear birth process) {X(t), t > O} is 
a birth-and-death process with A, = nA and py, = 0, for all n > 0. In this 
process, each particle gives birth to one particle, independently of others, and 
never dies. It is a pure birth process. 

(a) Suppose that initially the process is in state 1 (i.e., there is a single particle 
in the system). Show that the Kolmogorov forward equations (7.1) have the 
form P, (t) = —A P;(t) and P/(t) = (n—1)A Py_1(t) — nA P,(t), n = 2,3,..., 
with the boundary condition P,(0) = 1. 

(b) Verify that P,(t) =e~** (1 — re Naat n=1,2,.... Note that it isa 
geometric distribution that models the number of trials until the first success 
where the probability of a success is p = e~**. 

(c) Show that the mean of the Yule process at time t is E(X(t)) = e**, and 
the variance is Var(X (t)) = e*"(e*! — 1). 

(d) If A = 4 per week, what is the probability that there will be between 3 
and 5 particles at week 1? What is the mean and standard deviation of the 
number of particles at week 1? 


EXERCISE 7.2. Consider a Yule process, a birth-and-death process with pa- 
rameters A, = nA and py, = 0, for all n > 0. Suppose initially the process is 
in state m. That is, the initial size of the population is m particles. 

(a) Verify that the Kolmogorov forward equations (7.1) have the form 


P(t) = —mAPm(t) and P.(t) = (nm — 1)APa-i(t) — nAPalt), n = 
m,m+1,..., with the boundary condition P,,,(0) = 1. 
—1 n-—m 
(b) Verify that P,(t) = & emt (] —e*) ,n=mm+l,.... 
n—m 


Note that it is a negative binomial distribution of the number of trials until 
the mth success, where the probability of a success is p = e~**. 

(c) Show that the mean of the Yule process at time t is E(X(t)) = me’, and 
the variance is Var(X (t)) = me**'(e** — 1). 

(d) If there are originally 5 particles in the population and they multiply with 
rate \ = 0.2 per day, what is the probability that there will be exactly 12 par- 
ticles on day 2? What are the average and standard deviation of the number 
of particles on day 2? 


2Proposed in Yule, G. U. (1925). “A mathematical theory of evolution, based on the 
conclusions of Dr. J. C. Willis, F.R.S.” Philosophical transactions of the Royal Society of 
London. Series B, containing papers of a biological character, 213: 21-87. 
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EXERCISE 7.3. A linear death process {X(t),t > O} is a birth-and-death 
process with the initial state N, and parameters A, = 0 and py = ny for 
n=0,1,...,N—1. 

(a) Show that the probabilities P,,(¢), n = 0,1,...,N, satisfy the Kolmogorov 
forward equations 


Pi(t) = (n +1) Pasi(t) — np Pp(t), n=0,1,...,N—1. 


N n —n 
(b) Verify that P,(t) = ( Jers (1 Sue ,n=0,1,...,N, which 
nm 


is a binomial distribution with parameters N and p = e~“?. 

(c) Show that the mean and variance of this process at time t are E(X(t)) = 
Net and Var(X(t)) = Ne *(1 — e7#*). 

(d) Assume ps = 0.02 and N = 15. Find the probability that the process is in 
state 12 at time 3. What is the expected state at time 3? What is its standard 
deviation? 


EXERCISE 7.4. Consider a linear birth-and-death process {X(t), t > 0} de- 
scribed in Example 7.2. 

(a) Suppose A = 1.3 and yw = 0.2. Compute the probability that the process 
will be in state 4 at time 2. Find the mean and variance of the process at time 
2. 

(b) Simulate a 50-step trajectory of this process, assuming A = 1.3, w = 0.2, 
and the initial state is 1. 


EXERCISE 7.5. Consider an M/M/1 queue described in Application 7.1. 

(a) In the long run, how many customers do we expect to see in the system 
if \ > yw? Explain on an intuitive level. 

(b) Assume A = 3 and yu = 5. Find the probability that there will be more 
than 2 customers in the system in the long run. 

(c) Compute the average number of customers in the system in the long run. 
Use A = 3 and uw = 5. 

(d) Find the proportion of customers in the system in the long run who have 
to wait more than 1 minute. 


EXERCISE 7.6. Suppose the bird count in a flock can be modeled as a 
birth-and-death process with immigration and emigration. In this model, 
An = nA +a and tn = n+ 6 where a is the rate of immigration (join- 
ing the flock), and £ is the rate of emigration (leaving the flock). Generate 
a trajectory of the process until the flock size increases from 10 to 25 birds, 
assuming A= 1, a=0.3, 6 =0.1 and 
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units does each one span? Does the flock ever die out? 
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Branching Process 


8.1 Definition of Branching Process 


A discrete-time stochastic process {X,,n > 0} that gives the size of the 
nth generation of multiplying particles is called a branching process (or the 
Bienaymé-Galton- Watson process‘). It starts with Xo particles in the Oth gen- 
eration. Each particle survives for one-time unit, at the end of which it splits 
into a random number of particles with a known probability distribution. The 
offspring particles survive for one-time unit, and produce a random number 
of offspring, independently from each other, and the process continues. Put 
Xn 
formally, X, = S- Z;, where Z; is the size of the offspring of the ith par- 
i=1 
ticle in the (n — 1)st generation. The distribution of the offspring size Z; is 
identical for all particles, with p, = P(Z; =k), k =0,1,..., E(Z;) = py, and 
Var(Z;) = 0. 


PROPOSITION 8.1. A branching process is a Markov chain. 


PROOF: To prove that the Markov property holds, we use the fact that the 
distribution of the number of offspring for each individual particle is indepen- 
dent of the size of the current and all previous generations. We write 


PX = ain | Xo = JO; Xy — Jl; o nel = ee 
Jn=-1 


=P()> S25 | 56S fie See et Se) 


i=1 


1A French statistician Jules Bienaymé first addressed the problem of survival of family 
names in his 1845 article “De la loi de multiplication et de la durée des familles.” Société 
Philomatique de Paris. Extraits des Procés-Verbaux des Séances: 37 — 39. This problem 
was rediscovered later by Englishman Sir Francis Galton who posed the problem (under 
the number 4001) in The Educational Times and Journal of the College of Receptors, 
Vol. XXV, No. 143, on page 300. The solution by English mathematician Reverend Henry 
William Watson was published in the same journal, Vol. XXVI, No. 148, on page 115. 
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Jn-1 


t=1 


PROPOSITION 8.2. Consider a branching process with a single initial ancestor, 
Desa. 

(a) The mean of the size of the nth generation is E(X,,) = px”. 

(b) The variance is Var(Xo) = 0, Var(X1) = 07, and for n > 2, 


o? pnt (ae), if Al 


o2 


Var(Xn) = 
n, ifu=l. 


PROOF: (a) To find the expected value of X,,, we condition on the value of 
the size of the previous generation X,_1. We write 


(Xn) = E[E(Xn |Xn—1)) = a(S Zi|Xn-1)| 


= E(Xp-1 E(Z;)) = wE(Xn-1) = wE(Xp-2) = +++ = w"E(Xo) =p”. 


(b) The variance of Xo = 1 is 0. The variance of X1 = Z; is 0”. To compute 
the expression for the variance of X,, for n > 2, we condition on the value of 
Xn—1. We obtain 


Var(Xn) = Var |[E(Xn | Xn 1)] + E[Var(Xn | Xn-1)| 


Xn-1 
— Var | ( ye Zi|Xn-1)|] 
i=1 


Xn-1 
+ :[Var( De Zi|Xn-1)] = Var|Xn 1 1(Z;)] + [Xn 1Var(Z;)] 


= Var(uXn-1) + E(o? Xn-1) = pw? Var(Xn-1) + 0? p+ 
= p?[p? Var(Xn—2) +07 w"-?] +07 yp"! = p*Var(Xn_2) +07 (u"* +p") 
Sass Nas ss pe"—* Var(X1) a ei specs oh pPM—3) = a7 (ut A lh ee) 


t= n 
= ow tet pt) = 07 ph 7 if pA, 


and if 4 = 1, the variance is equal to a? n. 


EXAMPLE 8.1. (a) Suppose that the distribution of the size of the offspring is 
Bernoulli(4). We know that pw = s and a? = i. Therefore, the expected 
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size of the nth generation is E(X,) = (3)" with the standard deviation 
n— —(h)n n n 
Var(Xn) = 4/(2)(8)" | (GE) = GG - @)”)- 


(b) Assume now that the distribution of the offspring size is Binomial(2, 1/2). 
It means that pp = 1 and o? = 1/2. Hence, E(X,,) = 1, and the standard de- 


viation is \/Var(X. = /%. 


(c) Here we will assume that the distribution of Z;’s is Binomial(3, 1/2). 
Thus, the mean ys = 3/2 and variance is 0? = 3/4. The mean of the size of 
the nth generation, therefore, is found as E(X,,) = (3/2)” and the standard 


deviation is /Var(X,) = / (3)(3)" | (GS*) = V(D)"((H" - 0- 


Branching processes are classified according to the value of the mean size of 
the offspring. If wy < 1, the process is called subcritical, if ~ = 1, it is called 
critical, and if uw > 1, it is called supercritical. 


The probability of extinction is defined as the probability that a branching 
process with one initial ancestor will eventually have no particles. Put math- 
ematically, let 7 denote the probability of extinction. Then 


PROPOSITION 8.3. (a) For a subcritical (uw < 1) process, the probability of 
extinction is 1 (ao = 1). Intuitively, if not enough particles are being born, 
the population will surely go extinct. 

(b) For a critical (u = 1) process, 79 = 1 (so the population is guaranteed 
to become extinct), unless Z; = 1. In this case, the population consists of a 
single particle throughout all generations. 

(c) For a supercritical (2 > 1) process, the extinction can happen with a pos- 
itive probability, but this probability is less than 1. In fact, 79 is the smallest 
positive solution of the equation 


Co Co 
To = Se P(extinction| X; =k) p, = x 1h Dk. 


> 
ll 
fon) 
> 
ll 
oS 


Note that 79 = 1 is always one of the roots, which helps reduce this equation 
by one degree in case it has a polynomial form. 
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EXAMPLE 8.2. Going back to Example 8.1, we see that when the distribu- 
tion of the offspring size is Bernoulli(1/2) (u = 1/2) or Binomial(2, 1/2) 
(u = 1), the population will go extinct with probability 1. However, in the 
case of Binomial(3, 1/2), the extinction of the population is not a sure thing. 
It will happen with probability 7) where 7 is the smallest positive root of 
mo = me (1/2)? + 19 (3)(1/2)? + 78 (3)(1/2)? + 78 (1/2), or, rewritten in the 
standard form, 7g + 373 — 57 +1 = 0. Since 7 = 1 is a solution, the cubic 
equation is reduced to the quadratic one 7@ + 47 — 1 = 0, which roots are 
—2—\/5 and V5 — 2. The probability of extinction is the positive of the two 
roots, 7 = V/5 — 2 = 0.236068. 


EXAMPLE 8.3. Suppose a branching process {X,, n > 0} starts with a single 
particle and the particles multiply according to a geometric distribution with 
the probability mass function p(a) = p(1—p)*, x =0,1,2,..... 


(a) The mean of the geometric distribution is p = E(Z,,) = a The process 
is subcritical if <1, or <P <1, or p > 0.5. The process is critical if p= 1 
or p = 0.5. The process is supercritical if w > 1, or p < 0.5. 


1 re n 
(b) The average size of the nth generation is E(X,,) = uw” = (—*) . The 
Pp 


—P 
p2 


variance of the geometric distribution is 0? = , and hence, if p 4 0.5, 


tt =o at) = E(t 8y (EE 


_ 2 = EP 
1-p p Dp LL 
=p lS 0- EN) 
2p—1\ p p , 
1- 
pHs VaR Sota CEP 2m, 
Pp 


(c) To find the probability of extinction as a function of p,0 < p < 0.5, we 
note that 7 is the smallest positive solution of the equation 


Co Co D 
a ym p(1—p) PD, (nol p)) Rar 
This is a quadratic equation (79 — 1)(m — +) = 0. Therefore, 7 = = 


1—p —p 
For p > 0.5, m = 1. 
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8.2 Simulations in R 


SIMULATION 8.1. Here we simulate generation sizes for subcritical, critical, 
and supercritical processes. We assume that each population starts with a 
single initial particle and the respective offspring distributions are binomial 
with parameters n = 3, p = 0.2 (u = np = 0.6), n = 5, p = 0.2 (w = 1), and 
n= 3, p=0.6 (u=1.8). 


#subcritical branching process, mu=0.6 


k<- 1 
N<- cQ) 
N[1]<- 1 


set .seed (300168) 
for (i in 2:100) { 
N{i]<- sum(rbinom(N[i-1] ,3,0.2)) 
if (N[i]==0) { 
break } 


N 
[1] 12210 


#critical branching process, mu=1 
N<- c() 
N[1]<- 1 


set .seed (3554218) 
for (i in 2:100) { 
N[il<- sum(rbinom(N[i-1],5,0.2)) 
if (N[i]==0) { 
break } 


N 


[1] 1 3 6 610 6 7 23 3 2 1 
[13] 1 0 


#supercritical branching process, mu=1.8 
N<- cQ) 
N[1]<- 14 
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set.seed (965823) 
for (i in 2:20) 
N{i]<- sum(rbinom(N[i-1] ,3,0.6)) 


N 


[1] 1 2 3 7 13 23 
[7] 41 79 128 236 422 789 
[13] 1435 2609 4764 8594 15565 27964 
[19] 50574 91053 


From the above simulation, we see that the subcritical process dies out quickly, 
the critical takes a bit longer to die out, but the supercritical process grows 
to a large number of particles and doesn’t become extinct. 


SIMULATION 8.2. In this simulation, we generate and plot a trajectory of a 
branching process, in which offspring distribution is binomial with parameters 
n =3 and p=0.7. We utilize a self-referencing function in R and request to 
plot five generations of the process. The code is as follows. 


library (tidyverse) 


#specifying parameters gen.max<- 5 
prob<- 0.7 


#specifying seed 
set .seed (2443534) 


#simulating trajectory 

level.segment<- function(gen, y, branch.num) { 

branch <- data.frame(x=c(), y=c(), xend=c(), yend=c()) 
gen.remaining<- gen.max-gen-1 

if (gen.remaining < 0) return(branch) 


if (branch.num > 0) { 

branch<- rbind(branch, data.frame(x=gen, y=y, xend=gent1, 
yend=y), level.segment (gen=gen+1, y=y, branch.num=rbinom(1, 
3, prob))) 

} 


if (branch.num > 1) { 

branch<- rbind(branch, data.frame(x=gen, y=y, xend=gent1, 
yend=y+ 3“gen.remaining), level.segment (gen=gent+1, 
y=y+3"gen.remaining, branch.num=rbinom(1, 3, prob))) 


i: 
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Ss (branch.num > 2) { 
ranch<- rbind(br 
anch, dat 
yend=y-3" a eee ? a.frame (x= cm = 
Le Oe epi oe eG ee xend=gent1, 
-remaining, na Co pee Srne etree 
, 3, prob))) 


branch 


ib < iL 6 
p evel.segment (0, 0, rbinom(1 3 prob) 
2 2 ) 


seer trajectory 

plot (bpl, 1 

eae bp[,2], type="n", yaxt="n" 2 

process" ge(bp)), xlab="Generation" » xlim=<(0,5)5 
ss, panel.first=grid() no, ylab="Branching 


segments (bp[, 1] 
Det 21 brill pel) ) wd col—"b 
ca @ll= lue") 
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8.3 Applications of Branching Process 


APPLICATION 8.1. One of the research interests of cultural anthropologists is 
that of a long-term history of population dynamics. Suppose a certain geo- 
graphic area was initially populated by 25 families with 25 women of child- 
bearing age. In modern days, when a census of that area was taken, there were 
19,856 women of child-bearing age. Ten percent of them had one daughter, 
20% had two daughters, 60% had three daughters, and the others didn’t have 
any daughters. 


(a) From these data, we can estimate the mean size of female offspring. We 
have jt = (0)(0.1) + (1)(0.1) + (2)(0.2) + (3) (0.6) = 2.3. This is a supercritical 
process. The extinction of each family in this population is not going to hap- 
pen for sure. In fact, the estimated probability of extinction of each family 7 
solves 7p = 0.1+ 0.17% + 0.272 + 0.678, or, equivalently, 672 + 87% — 1 =0. 
The solution is 7p = 22-4 = 0.115069. The estimated probability that all 
the 25 families become extinct is 72° which is very close to zero. However, the 
probability that at least one of the 25 families becomes extinct is computed 
as 1 — (1 — 0.115069)?° = 0.952931. 


(b) From the data, we can also assess how long ago the initial settle- 
ment took place. We know that on average, the size of the nth genera- 
tion is (25)(7”) = (25)(2.3)". Therefore, (25)(2.3)” = 19,856. From here, 
n = In(19, 856/25) /In(2.3) = 8.0169 or about eight generations. Assuming 
that it takes around 25 years for a generation to mature, we can say that the 
settlement was established about (8)(25) = 200 years ago. 


APPLICATION 8.2. In epidemiology, the most basic model of the spread of an 
infectious disease is a branching process. An initially infected individual will 
either recover, for instance, with probability 0.1 without infecting others, or 
will infect a zero-truncated Poisson(A) random number of individuals where 
A = 2.4, say. 


(a) To find the mean of the nth generation of infected individuals, we 
note that the probability mass function for the offspring is p(0) = 0.1 and 


ae ee (2.4)" e 34 
p(n) = (0.9) fo = (0.9) nl T2ee8: n= LDS 33 cats The mean 
.9)(2.4 
of this distribution is up = (0.9) - wee) = 2.375501. The 


1—e- 1—e-24 
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Xr” a 0.9 
Ae e€ — (P= SA EN) KS 


Co 
* ® ed 2 
variance is o~ = (0.9 ) n 
(0-2) n! 1—e-> 


n=1 


a (2.4 + (2.4)?) — (2.375501)? = 2.433697. 


= e-2-4 


(b) The expected number of infected individuals in the nth generation is 
pe” = (2.375501), and the standard deviation is 


i= Le 


T— Ti 
2,.n-1 = n—-1 
op ( ) (2.433697) (2.375501) ( Se 


1—p 
= 4/ (1.769317) (2.375501)"—1((2.375501)" — 1). 


(c) Since « > 1, this is a supercritical process. The probability 7 that the 
infection stops spreading is the smallest positive solution of the equation 


6 oo h (2.4)” e724 
70 = (0.1) (70) + a To (0.9) nm! 1—e-2-4’ 
or 
0.9  (2.470)” OO: -Pigae 
e014 aay ON gaz (2-1). 


Solved numerically, 7 = 0.1340992. The lines of code that produce this an- 
swer follow. 


library (rootSolve) 

equation<- function(x) 
x-0.1-0.9/(exp(2.4)-1)*(exp(2.4*x)-1) 
uniroot.all(equation, c(0,0.99)) 


0.1340992 


a 


Exercises 


EXERCISE 8.1. Consider a colony of bacteria. Bacteria are known to repro- 
duce asexually by binary fission, splitting into two identical cells. Suppose at 
time 0 the colony size is X9 = 100 bacteria. Suppose also that at the end of a 
time unit, each bacterium, independently of the others, dies with probability 
0.25, splits into two with probability 0.6, or continues living with probability 
0.15. 
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(a) Show that the colony growth can be modeled as a supercritical branching 
process. Find the expected size of the nth generation and its variance. 

(b) Compute the extinction probability for descendants of each bacterium. 
(c) Find the probability that descendants of at least one of ten bacteria go 
extinct. 


EXERCISE 8.2. Consider a branching process with a sole ancestor and a 
Poisson(A) proliferation distribution. 

(a) Determine the values of A for which this process is supercritical (critical, 
subcritical). 

(b) Give expressions for the mean and variance of the size of the nth genera- 
tion. 

(c) Give the equation that the extinction probability solves. Plot a graph of 
the numeric solution as a function of A > 1. 


EXERCISE 8.3. Based on work by Alfred J. Lotka? who analyzed the data 
from the 1920 U.S. Census, suppose that male offspring has a zero-adjusted 
geometric distribution of the form: 


p(0) = 0.4828 and p(n) = (0.228292)(0.5586)""! if n=1,2,3,.... 


(a) Find the expected size of male offspring and its standard deviation. 

(b) Consider a single male ancestor. Find the expected size of the nth gener- 
ation of his descendants and its standard deviation. 

(c) Compute the probability of extinction. 


EXERCISE 8.4. Parlaying in gambling is defined as a series of bets in which 
winnings are used as a stake for further bets. This process can be modeled as 
a branching process. Suppose a gambler starts with a stake of $1, and can win 
$1 with probability 0.3, or $15 with probability 0.2, or $20 with probability 
0.1, or $0 with probability 0.4. 

(a) Show that this is a supercritical process. 

(b) Find the expected winnings on the fifth bet. 

(c) Find the probability that the gambler’s stake eventually turns into $0. 


EXERCISE 8.5. The spread of computer viruses is often modeled as a branch- 
ing process. Assume that initially one computer is infected with a virus, and 
every day the virus is sent to other computers which number has a discrete 
uniform distribution between 0 and 3. 


2Lotka, A. J. (1931a). “The extinction of families, I.” Journal of the Washington Academy 
of Sciences, 21(16): 377 — 380; and Lotka, A. J. (1931b). “The extinction of families, II.” 
Journal of the Washington Academy of Sciences, 21(18): 453 — 459. 
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(a) Is this process subcritical, critical, or supercritical? 

(b) What are the average number and standard deviation of infected com- 
puters on day 10? 

(c) Will the spread of viruses die out with probability 1? If not, find the ex- 
tinction probability. 

(d) Simulate the number of infected computers during 10 days. What is the 
simulated total number of infected computers? 


EXERCISE 8.6. Consider a branching process that starts with a single particle 
in generation 1. Assume that the offspring has a distribution with the proba- 
bility mass function p(0) = 0.1, p(1) = 0.4, p(2) = 0.5. 

(a) Generate the size of the first 20 generations of this process. What is the 
size of the offspring of the 19th generation? How many total particles are in 
this population? 

(b) Simulate and plot a sample trajectory of this process for the first six gen- 
erations. 
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Brownian Motion 


9.1 Definition of Brownian Motion 


A stochastic process { B(t), t > 0} is called a standard Brownian motion! (or 
a Wiener process”) if: (i) B(O) = 0, (ii) it has independent and stationary 
increments, and (iii) B(t) ~ N(0, t), t > 0. 


REMARK 9.1. Brownian motion is sometimes termed the Bachelier process. 
In his 1900 paper® a French mathematician Louis Bachelier derived the Brow- 
nian motion as the limit of random walks. 


PROPOSITION 9.1. (a) Brownian motion is everywhere continuous but 
nowhere differentiable. 
(b) Brownian motion hits every real number infinitely many times. 


PROOF: The proofs of both statements are omitted. 


PROPOSITION 9.2. For a standard Brownian motion, the covariance between 
B(s) and B(t) is Cov(B(s), B(t)) = min(s,t), s,t> 0. 


1English botanist Robert Brown observed the movement of dust particles in liquid and 
described the motion in Brown, R. (1828). “A brief account of microscopical observations 
made in the months of June, July and August, 1827, on the particles contained in the pollen 
of plants; and on the general existence of active molecules in organic and inorganic bodies.” 
Philosophical Magazine, Series 2, 4: 161 — 173. 

2Norbert Wiener proposed a rigorous mathematical model for a Brownian motion in 
Wiener, N. (1921). “The average of an analytic functional and the Brownian movement.” 
Proceedings of the National Academy of Sciences of the United States of America, 7(10): 
294 — 298. 

3Bachelier, L. (1900). “Théorie de la spéculation.” Annales Scientifiques de lV’Ecole Nor- 
male Supérieure, 17: 21 — 86. 
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PROOF: Suppose s < t. Since the mean of a Brownian motion is zero and its 
increments are independent, we obtain 


Cov(B(s), B(t)) = E(B(s)B(t)) = E[B(s)(B(t) — B(s) + B(s))] 
= E[B(s)(B(t) — B(s))] + E(B(s))” = E(B(s))E(B(é) — B(s)) + E(B(s))” 


PROPOSITION 9.3. (RESCALING RELATION). For any real a, B(at) and 
/aB(t) have the same distribution, where B(t) denotes a standard Brow- 
nian motion. 


PRooF: The distribution of B(at) is N(0,at) by the definition of a Brow- 
nian motion. Also, since B(t) ~ N(0,t), we have that /aB(t) is also nor- 
mal with mean E(,/aB(t)) = /aE(B(t)) = 0 and variance Var(./aB(t)) = 
aVar(B(t)) = at. 


EXAMPLE 9.1. Suppose we want to compute the conditional probability that 
a standard Brownian motion is below 3 at time 3, given that it is equal to 1 
at time 1. We write 


P(B(3) < 3| B(1) = 1) = P(B(3) — B(1) < 3-1] BA) = 1) 
= P(B(3) — B(1) <2) (by independence of increments) 
(by stationarity of increments) 
= P(V2B(1) <2) (by the rescaling relation) 
= P(B(1) < V2) = &(V2) = 0.92135. 


Here and later ®(-) denotes the standard normal cumulative distribution func- 
tion. 


PROPOSITION 9.4. (DISTRIBUTION OF HITTING TIME). Denote by T;, the 
first time a standard Brownian motion hits a level a > 0. The cumulative 
distribution function of T, is Fp,(t) = 2(1 — ®(a/V)), t > 0. 


PROOF: We write P(B(t) > a) = P(B(t) > alT™ < OPM < O+ 
P(B(t) > alT, > t)P(T, > t). If Ty > t, B(t) hasn’t reached the level 
a yet, and it is impossible to have B(t) > a. Thus, the second term is equal 
to 0. 


Definition of Brownian Motion 155 


Also, from symmetry (see the picture), the Brownian motion is as likely to go 
up after hitting a as down, therefore, 


P(B() > a|T < HY =P(BO < alt, <== 


B(t) 


Consequently, P(B(t) > a) = $P(T. < t), and hence, P(T, < t) = 
2P(B(t) >a) = 2(1 — ®(a/V2)). 


REMARK 9.2. In the proof above, we used the symmetry property of a Brow- 
nian motion. Put rigorously, this property is known as the reflection principle 
and is stated as: if a path of a Brownian motion reaches a value B(s) at time 
s, a path after time s has the same distribution as its reflection about the 
value B(s). 


EXAMPLE 9.2. The probability that a Brownian motion reaches level 1 by 
time 5 is P(T, <5) = 2(1 — &(1/V5)) = 0.6547. 


PROPOSITION 9.5. (DISTRIBUTION OF MAXIMUM VALUE). Denote by M(t) 
the maximum value that a Brownian motion attains on the interval [0, t]. The 
cumulative distribution function of M(t) is Fry)(a) = 2®(a/Vt) — 1, for 
a> Oandt > 0. 


Proor: Note that M(t) > a> 0, if and only if T, < t. Therefore, 


P(M(t) < a) =1—P(M(T) > a) =1-P(T, <2) 
=1-2(1 — ®(a/vt)) = 2®(a/Vt) — 1. 
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EXAMPLE 9.3. The probability that a Brownian motion is below 2 everywhere 
on the interval [0, 4] is P(M(4) < 2) = 26(2/V/4) —1 = 20(1)-—1 = 
0.682689. 


9.2 Processes Derived from Brownian Motion 

9.2.1 Brownian Bridge 

A Brownian bridge is a stochastic process {X (t), 0 < t < 1} that satisfies the 
following properties: (i) X(t) is normally distributed, (ii) X(0) = X(1) = 0, 
(iii) E(X(t)) = 0, (iv) Var(X(t)) = t(1 — £), and (v) Cov[X(s), X(t)] = 
min(s,t) — st, 0<s,t<1. 


We can think of a Brownian bridge as a Brownian motion on the interval [0, 1], 
tied at the two ends. Note that the variance is equal to 0 at both ends of the 
interval as it should be since the values are deterministic at those points, and 
increases toward the middle, reaching its maximum at t = 1/2. 


More generally, a Brownian bridge on the interval [0,7], {X(t), 0<t< T}, 
is such that: (i) X(t) is normally distributed, (ii) X(0) = X(T) = 0, (iii 
E(X(t)) = 0, (iv) Var(X(t)) = (1 — 4/T), and (v) Cov[X(s),X()] = 


min(s,t) — st/T, 0< s,t<T. 


0} is a standard Brownian mo- 


PROPOSITION 9.6. Suppose that {B(t), t > 
), O<t< 1. Then {X(#),0<t< lisa 


tion, and let X(t) = B(t) —tB 
Brownian bridge. 


PRooF: X(t) has a normal distribution because B(t) is normally distributed. 
The mean of X(t) is E(X(t)) = E(B(t)) — tE(B(1)) = 0 — (t)(0) = 0. Now, 
recall that Cov(B(s), B(t)) = E(B(s)B(t)) = min(s,t) (see Proposition 9.2). 
Assuming s < t, we compute the covariance between X(s) and X(t) as 


Cov(X(s), X (t)) = E[B(s) — sB(1), B(t) — tB(1)] 
= E(B(s)B(t)) — tE(B(s)B(1)) — sE(B(1) B(t)) + stE(B(1))? 
= min(s,t) — tmin(s, 1) — s min(1,t) + stVar(B(1)) 


=s—ts—st+st=s-— st =min(s,t) — st. 
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REMARK 9.3. The proof of the above proposition can be extended (do it!) to 
show that {X(t) = B(t)— 4 B(T), t > 0} is a Brownian bridge on the interval 
(0, 7). 


9.2.2 Brownian Motion with Drift and Volatility 


Let {B(t), t > 0} denote a standard Brownian motion. A stochastic process 
{X(t) = pt + o Bit), t > O} is called a Brownian motion with the drift 
coefficient and volatility (or diffusion) coefficient o. 


PROPOSITION 9.7. The distribution of X(t) is normal with mean jt and vari- 
ance a” t. Also, the covariance between X(s) and X(t) is 0? min(s, t). 


PRooF: The distribution of X(t) is normal since B(t) is normally distributed. 
The mean of X(t) is E(X(¢)) = wt + o E(B(t)) = wt, and the variance is 
Var(X(t)) = Var(ut + o B(t)) = o?Var(B(t)) = 0? t. The covariance be- 
tween X(s) and X(t) is Cov(X(s), X(t)) = a((ns +0 B(s))(pt+ o B(t))) = 
i(.9 +0 B(s))E(ut +0 B(t)) = (45)(ut) + 0?E(B(s)B(t)) — (s)(ut) = 


o” min(s, t). 


9.2.3. Geometric Brownian Motion 


A stochastic process {Y(t) = Y(0)exp(ut + o B(t)),t > O} is called a 
geometric (or exponential) Brownian motion. 


PROPOSITION 9.8. The distribution of Y(t) is log-normal with the density 
function 


7 1 (Iny —InY(0) — pt)? 
fyi y) = inet ey exp ( to). y > 0. 


The mean and variance are 


i(Y (t)) = Y (0) elttort/2 and Var(Y (t)) = [Y (0)]2e2## + o°t(e97t - 1). 


PROOF: Since B(t) ~ N(0,t), the cumulative distribution function of Y(t) is 
derived as follows: 


Fyw(y) =P(Y(t) sy) = P(ut + o BY) < my-InY(0)) 


=P(BO) < my VO) ~ Ht) 9A). y > 0. 
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The density is 


1 (ny —InY(0) — pt)? 
= Fy) === ow (- S SO) , 
fri) (y) Y (t) (y) Jone ty exp V2 t > Yo 0 
Using the expression for the moment generating function of B(t) ~ N(0, t), 
E(e7 B®) = er ye, we get that the mean of Y(t) is 


E(Y (t)) = E(Y (0) hr ree) = Y(0) eltt i(e7 BO) = y(0) cut ort/2 


and the variance is 


Var(Y (t)) = E(Y(t))? — [E(Y (t))]° = E(Y(0) ett +o Bt)? 
= (Y (0) gees spy 


= [Y(0)]? e7#* (er Bt) = [Y (0)]? e2#t tot = Y (0)]2 e2et +207 


= [y (0)? e2uttort 
= OP eer er ta): 


9.2.4 The Ornstein-Uhlenbeck Process 


The Ornstein-Uhlenbeck process* { X(t), ¢t > 0} is a stochastic process of the 
form 


X(t) = X(0)e—** + w(1—e7** fe eT one ere); 
(t) = X(O)e*" + (1 — 8) + eer P*B(e#* 1) 
Here wy is the drift, 0 > 0 is the volatility, and @ > 0 is an additional param- 


eter. 


PROPOSITION 9.9. The mean of X(t) is X(0)e~°* + (1 — e~®*), and the 


2 
: aS = 
variance is ve —e cae 


PROOF: The mean of X(t) is 


i(X(t)) = X(O)e9* + p(1 — e-**) Sh ee E(B(c?*" — 1)) 
— xij" +p(1—e78), 


4First appeared in Uhlenbeck, G. E. and L. S. Ornstein (1930). “On the theory of Brow- 
nian motion.” Physical Review, 36: 823 — 841. 
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since the expected value of a Brownian motion (in this case, a time- 
transformed Brownian motion) is equal to 0. The variance of X(t) is 


o? a 
Var(X(t)) = me Var( Be" _ 1)) _ aed Cal _ 1) 
2 
_ F 7, 26+ 
= oes e€ ). 


REMARK 9.4. Note that as t increases, the mean of the Ornstein-Uhlenbeck 
process tends to yw. Therefore, the drift yz is the long-term mean, and the pro- 
cess is called mean-reverting. The parameter @ represents the rate by which 
the process reverts towards the mean. In addition, the variance of this pro- 
cess is bounded by a constant o7/(20), and in the long run, approaches this 
constant. 


a 


9.3. Simulations in R 


SIMULATION 9.1. (ONE-DIMENSIONAL STANDARD BROWNIAN MOTION). The 
code below simulates three trajectories of a standard Brownian motion on the 
time interval that has 500 increments of size 0.01. The plot follows. 


BM<- matrix(NA, nrow=500, ncol=3) 


#specifying seed 
set .seed (8221056) 


#simulating trajectories 
or (Cj iim igs) a 
BM[1,j]<- 0 


for (i in 2:500) 
BM[i,jl]<- BM[i-1,j] + sqrt(0.01)*rnorm(1) 
} 


#plotting trajectories 

matplot(BM, type="1", lty=1, lwd=2, col=2:4, 
ylim=c(range(BM)), xlab="Time", ylab="Brownian motion", 
panel.first=grid()) 
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ede cce 


Brownian motion 


SIMULATION 9.2. (TWO-DIMENSIONAL BROWNIAN MoTIon). A 
two-dimensional Brownian motion is a stochastic process that keeps track of 
two coordinates, both of which are independent Brownian motions. The R 
syntax below simulates and plots one trajectory of a two-dimensional Brown- 
ian motion. 


BM<- matrix(NA, nrow=5000, ncol=2) 


#specifying seed 
set .seed (34885002) 


#simulating two independent Brownian motions 
rere (Cj aim ie2) fl 
BM[1,j]<- 0 


for (i in 2:5000) 
BM[i,j]<- BM[i-1,j] + sqrt(0.01)*rnorm(1) 
L 


#plotting trajectory 
plot (x=BM[,1], y=BM[,2], type="1", col=4, xlab="x", ylab="y", 
xlim=range(BM[,1]), ylim=range(BM[,2]), panel.first=grid() ) 
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#adding starting point 
points(cbind(BM[1,1], BM[1,2]), pch=16, cex=2, col="green") 


#adding ending point 
points (cbind(BM[5000,1] ,BM[5000,2]), pch=16, cex=2, 
col="red") 
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SIMULATION 9.3. (THREE-DIMENSIONAL BROWNIAN Motion). A three- 
dimensional Brownian motion is a stochastic process that models position 
by three coordinates, defined by three independent Brownian motions. Below 
we simulate and plot a single trajectory of a three-dimensional Brownian mo- 
tion. 


nsteps<- 2000 
BM<- matrix(NA, nrow=nsteps, ncol=3) 


#specifying seed 
set .seed (1133205) 
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#simulating three independent Brownian motions 
ieee (Gj alin 38) af 
BM[1,j]<- 0 


for (i in 2:nsteps) 
BML[i,j]<- BM[i-1,j] + sqrt(0.01)*rnorm(1) 
} 


#plotting trajectory 

library (plot3D) 

lines3D(BM[,1], BM[,2], BML,3], col=4, xlab="x", ylab="y", 
zlab="z", xlim=range(BM[,1]), ylim=range(BM[,2]), 
zlim=range(BM[,3]), bty="b2", ticktype="detailed") 


#adding starting point 
points3D(x=BM[1,1], y=BM[1,2], z=BM[1,3], add=TRUE, pch=16, 
cex=2, col="green") 


#adding ending point 
points3D(BM[nsteps,1], BM[nsteps,2], BM[nsteps,3], add=TRUE, 
pch=16, cex=2, col="red") 
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SIMULATION 9.4. (BROWNIAN BRIDGE). The following code generates three 
trajectories of a Brownian bridge. First we simulate three trajectories of a stan- 
dard Brownian motion { B(t), t € [0,500]}, and then turn them into Brownian 
bridge trajectories by computing { X(t) = B(t) — =a B(500), t € [0, 500}} (see 
Remark 9.3). The graphical output is given below. 


#defining Brownian motion and Brownian bridge as matrices 
BM<- matrix(NA, nrow=500, ncol=3) 
BB<- matrix(NA, nrow=500, ncol=3) 


#specifying seed 
set .seed (76435567) 


#simulating trajectories of Brownian motion 
for (j in 1:3) { 
BM[1,j]<- 0 


for (i in 2:500) 
BML[i,j]<- BM[i-1,j] + sqrt(0.01)*rnorm(1) 
} 


#computing trajectories of Brownian bridge 
pixel Gye Ma ECD I 

for (i in 1:500) 

BB[i,j]<- BM[i, j]-i/500*BM[500,j] 

is 


#plotting trajectories of Brownian bridge 

matplot (BB, type="1", lty=1, lwd=2, col=2:4, 
ylim=c(range(BB)), xlab="Time", ylab="Brownian bridge", 
panel. first=grid()) 
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Brownian bridge 


Bid eeesceeemeeeh ace 


(ote eee eee eet 


SIMULATION 9.5. (BROWNIAN MOTION WITH DRIFT AND VOLATIL- 
ITY). Below we generate three trajectories of a Brownian motion with drift 
pi = 1.3 and volatility 0 = 0.5. The code and plot follow. 


#specifying parameters 
mu<- 1.3 
sigma<- 0.5 


#defining Brownian motion as matrix 
BM<- matrix(NA, nrow=500, ncol=3) 


#specifying seed 
set .seed (8463338) 


#simulating trajectories 
for (j in 1:3) { 
BM[1,j]<- 0 
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for (i in 2:500) 
BM[i,j]<- mu*0.01+BM[i-1,j] + sigma*sqrt (0.01) *rnorm(1) 
a: 


#plotting trajectories 

matplot(BM, type="1", lty=1, lwd=2, col=2:4, 
ylim=c(range(BM)), xlab="Time", ylab="Brownian motion with 
drift and volatility", panel.first=grid()) 
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SIMULATION 9.6. (GEOMETRIC BROWNIAN MOTION). Here we simulate 
three trajectories of a geometric Brownian motion with the initial value 
Y(0) = 2, the drift coefficient 4 = 1.3, and volatility o = 0.5. Since the 
trajectories of a Brownian motion with these values of drift and volatility 
have already been constructed in Simulation 9.5, all we need to do is to apply 
the exponential function to the simulated trajectories and multiply by the 
initial value. The code and graph follow. 


#computing trajectories of geometric Brownian motion 
GBMO<- 2 
GBM<- GBMO*exp (BM) 


#plotting trajectories 

matplot(GBM, type="1", lty=1, lwd=2, col=2:4, 
panel.first=grid(), ylim=c(range(GBM)), xlab="Time", 
ylab="Geometric Brownian motion", panel.first=grid()) 
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SIMULATION 9.7. (THE ORNSTEIN-UHLENBECK PROCESS). We base our sim- 
ulation of a trajectory of an Ornstein-Uhlenbeck process {_X (t), t > 0} on the 
approximate recursive difference equation that it satisfies: 


X(t + At) = X(t) + O(u— X(t)) At + oVAtB(1) 


where At > 0 denotes a small increment of t. We omit the proof of this for- 
mula as it involves tedious algebra. 


We use this relation with time increments At = 1 to simulate a trajectory for 
the values of the parameters X(0) = 2,6 = 0.8, » = 1.6, and o = 0.5. The 
code and graph follow. 


#specifying parameters 


theta<- 0.8 
mu<- 1.6 
sigma<- 0.5 


#specifying seed 
set .seed (2043442) 


#defining Ornstein-Uhlenbeck trajectory as vector 
OU<- c() 


#specifying initial value 
OU[1]<- 2 
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#simulating trajectory 
for (i in 2:100) 
OULi]<- OU[i-1]+theta* (mu-OU[i-1])+sigma*rnorm(1) 


#plotting trajectory 
plot(1:100, OU, type="1", 1lty=1, lwd=2, col=4, xlab="Time", 
ylab="0rnstein-Uhlenbeck process", first.panel=grid() ) 
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9.4 Applications of Brownian Motion 


APPLICATION 9.1. Animal behavior researchers use the Brownian bridge to 
model movements of herds as they walk on their trails during daylight time and 
return to their designated lodging for an overnight stay. Suppose researchers 
observe the movements of a herd of deer during 8 hours of daylight. The main 
goal of the research is to estimate the distance between the north-most and 
south-most points that the deer have reached. This distance approximates the 
diameter of the deer home range. Assuming that the unit of measurement is 
one-tenth of a mile, below we compute the theoretical mean diameter of the 
home range in miles, and then simulate 1,000 trajectories to give an empirical 
estimate. 
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(a) It can be shown (see Exercise 9.9) that the expected value of the maxi- 


mum of a Brownian bridge on the interval [0, 7] is 4 / uate From symmetry, 


it can be argued that the minimum is expected to be of the same magnitude 
but with a negative sign. Therefore, the expected diameter of the deer home 


range is x / ar - ( - x / =r) = ar We will assume that T is given in 


minutes, and thus, T = (8)(60) = 480 minutes. Hence, the theoretical value 


of the mean diameter of the home range is (480) = 27.45873 tenths of a 


mile or 2.75 miles. 


(b) The code below simulates 1,000 trajectories of a Brownian bridge on the 
time interval [0,480] with an increment step of 1, and computes the sample 
mean of the range for the simulated trajectories. 


#defining Brownian motion and Brownian bridge as matrices 
BM<- matrix(NA, nrow=480, ncol=1000) 
BB<- matrix(NA, nrow=480, ncol=1000) 


#specifying seed set.seed (6769712) 


#simulating trajectories of Brownian motion 
for (j in 1:1000) { 
BM[1,j]<- 0 


for (i in 2:480) 
BM[i, j]<- BM[i-1,j] + rnorm(1) 
} 


#computing trajectories of Brownian bridge 
for(j in 1:1000){ 
for (i in 1:480) 
BB[i,j]<- BM[i, j]-i/480*BM[480, j] 
c 


#computing ranges 

range<- c() 

for(j in 1:1000) { 

range[j]<- max(BB[,j])-min(BB[,j]) 


#computing sample diameter of home range 
print (diameter<- mean(range) ) 


26 .80793 


Thus, the sample diameter of the home range is 26.80793/10 = 2.68 miles. 
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APPLICATION 9.2. A geometric Brownian motion is often used to model 
the behavior of a stock price over time. The data set downloaded from 
https://finance.yahoo.com/quote/AMZN/history/ contains Amazon.com, Inc. 
daily stock prices at the closing time of stock market exchange between 
01/02/2020 and 06/30/2021, a total of 377 business days. First, we plot the 
data. 


stock.data<- read.csv(file="./AMZN.csv", header=TRUE, 
aep=".") 


date<- as.POSIXct(stock.data$Date) 
price<- stock.data$Close 


#plotting stock price against date 
plot(date, price, type="1", lwd=2, cex=0, col="light blue", 
xlab="Time", ylab="Stock price", first.panel=grid() ) 
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Now we estimate the parameters and simulate a trajectory of a geometric 
Brownian motion. The model for the stock price is 


{X(t) = X(ti) exp(ut +o B(t)), th <t < ts77}. 


We express the increments of the natural logarithm of the process as 


In X(t;) —In X(tj_-1) =n cara = p(t; — ti-1) + 0 (B(ti) — B(ti_1)). 
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We take the time increments of unit length, t; — t;-; = 1, and argue using 
the stationarity of increments that the log-ratios (or log-price increments) are 
distributed as 4+0 B(1). That is, they have a normal distribution with mean 
y. and variance 07. The code given below plots a histogram, estimates yz and 
o by the sample values, simulates trajectories of the geometric Brownian mo- 
tion, and plots the actual and simulated prices on the same graph. 


#calculating increments of log-price 
log.inc<- c() 


pricel<- price[-1] 
pricel.lag<- head(price, -1) 
log.ratio<- log(pricel/pricel.lag) 


#plotting histogram 

library (rcompanion) 

plotNormalHistogram(log.ratio, xlab="Log-price increments", 
col="light blue") 
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From the histogram, a bell-shaped curve reasonably describes the density of 
the log-price increments, thus we conclude that the distribution can be as- 
sumed normal. 


#estimating parameters 
print (mu. hat<- mean(log.ratio)) 


0.001581681 


print (sigma.hat<- sd(log.ratio) ) 
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0.02156408 


#specifying Brownian motion as vector 
BM<- c() 


#specifying initial value 
BM[1]<- 0 


#specifying seed 
set .seed (43567347) 


#simulating Brownian motion with drift and volatility 
for (i in 2:377) 
BM[i]<- mu.hat+BM[i-1] + sigma.hat*rnorm(1) 


#computing values for geometric Brownian motion 
GBM<-price[1] *exp (BM) 


#plotting actual and simulated trajectories 

plot(date, price, type="1", lty=1, lwd=2, col="blue", 
xlab="Time", ylab="Stock price", first.panel=grid() ) 
lines(date, GBM, lwd=2, col="green") 

legend("bottomright", c("Actual price", "Simulated price"), 
lty=1, col=c("blue", "green") ) 
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APPLICATION 9.3. A geometric Brownian motion has another very famous 
application in the financial world. In 1997, two American economists Myron 
Scholes and Robert Merton were awarded the Nobel Prize in Economics for 
the Black-Scholes-Merton Option Pricing model. In 1973, Fischer Black and 
Myron Scholes® published the derivation of the model and Robert Merton® 
expanded the results. Black died in 1995, so he wasn’t awarded the Nobel 
Prize, for it is not given posthumously. 


In this application, we discuss the model and derive the final formula. First, 
we introduce the key concepts. 


In the financial market, a stock option is the right to buy (or sell) a stock at 
a predetermined price at a fixed time in the future. 


An individual can buy (or sell) the stock at the price X(s) at time s < t, and 
then sell (or buy) the stock at time t for the price X(t). Suppose also that the 
individual can buy (or sell) at time 0 a stock option that gives him the right 
to buy a stock at time t for the price K per share. How much should he pay 
for one share of the stock option? 


Suppose we loan out $1 today with a risk-free interest compounded contin- 
uously at the fixed rate r. Then by time t, it would grow into amount $e"*. 
From here, we conclude that a $1 at time ¢ is worth $e~™ in today’s money. 
The rate r is termed the discount factor, and the function e~" is called the 
discount function. It represents the present value of an amount of $1 at time t. 


We will assume that it is a fair market and there is no opportunity for an 
arbitrage. That is, there is no opportunity for a sure profit. Under this as- 
sumption, the expected return of buying (selling) one share of stock at time 
s < tand selling (buying) it at time t is zero, which translates into the identity 
involving the present values of the stock prices, 


u[e7"’ X(t)| X(u), OS u<s] =e7"* X(s). (9.1) 


Turning to the stock option, if the price of one share of stock at time t is below 
K, it is not reasonable to excise the option. Therefore, the present value of 
the option is e~"'(X(t) — K), if X(t) > K, and 0, otherwise, which can be 
written as e~"' (X(t) — K)*. 


5Black, F. and M. Scholes (1973). “The pricing of options and corporate liabilities.” 
Journal of Political Economy, 81(3): 637 — 654. 

6Merton, R. (1973). “Theory of rational option pricing.” Bell Journal of Economics and 
Management Science, 4 (1): 141 — 183. 
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Let C' be the price of one share of option at time zero. This is the quantity 
that we need to determine. In order not to create an arbitrage opportunity, 
we must have 


Ele" (X(t) — K)t — C] =0, or C = ec" E[(X(t)— K)t]. (9.2) 


The Black-Scholes-Merton model assumes that X(t) is a geometric Brown- 
ian motion { X(t) = X(0)e“'+7?©, ¢ > 0} with the drift parameter p: and 
volatility o. First, we see under what condition { X(t), t > 0} satisfies (9.1), 
and then plug this process into (9.2) to derive the final expression for C. 


Using the independence and stationarity of increments of the Brownian mo- 
tion and utilizing the expression of its moment generating function, we write 


Ble""* X(t) | X(u), OS u<s] =e7"* X(0) nfe#? +7 BM | Biu), O<uK<s] 


= e~"*X(0) elt o B(s) i et@ POF) = gor X(s) elt 8) ner 3) 


=e"! X(s) elt(t-9)4 (ts) _ ert (ute? /2)4— (ute? /2)6 4 (5) = e** X(s), 


if and only if w + o?/2 = r. This is the sought-for condition on the process 
{ X(t), t > 0}. We now use it in the expression (9.2). We compute 
1 22 
C=e"'E|(X(t)-K)t =e f X(0) ett ¢Vie_K)* —_e-* az. 
(x-Ky*] P(x) Ve 

We want X(0) e#!+7v** — K > 0, so z > (In(K/X(0))—pt)/(o V4). Denote 
by A = (wt — n(K/X(0)))/(o Vt). Then the lower limit of integration is — A. 
We continue 


oe 1 22 
C= xX(0 eet elttovtiz_-_ o- dz 
(0) > om 


aml 22 
—rt 22 
—e K ——e 7 dz 
i V2 


ae | zo Vi)? Se? UL 2 
= xo | ey ae eee otk | e = dz 
_A V2T _a V2r 


Ato Vt 1 2 . A 1 

-Fdz—e"tK ——e Td 
Noe gg Care ie Teo 
= X(0)6(A + ovt) — e"' K OA). 


To work with a numeric example, suppose the current price of one share of 
a stock is X(0) = $100. Suppose the stock price can be modeled by the 
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Black-Scholes-Merton model with the drift coefficient pp = —0.45 and volatil- 
ity 0 = 1.1. We want to compute the cost of the option to buy one share of 
the stock at time t = 2 for the cost of K = $120. We write 


_ ot — n(K/X(0)) _ (0.45) (2) — In(120/100) 
ir oe ee 0.69574, 
r=pt x - 0.454 2" — 9.55, 


and 


C = X(0)®(A + ovt) — e-"' K GA) 
= (100)®(—0.69574+ (1.1) V2) — e~ -155)) (120) &(—0.69574) = $59.09. 


APPLICATION 9.4. As opposed to stock and option prices that can rise in- 
definitely, interest rates and commodity prices move in a limited range. If 
their values are high, the demand drops, and consequently, the values drop. 
Likewise, if the values are low, demand increases, and eventually, the values 
increase. This characteristic is called a reversion to a long-run mean. 


The Ornstein-Uhlenbeck (OU) process is a good mathematical model that 
captures this mean-reversion property. Below we fit the parameters of the OU 
process to a publicly available data set on daily natural gas prices between 
1/4/2010 and 8/11/2020 (downloaded from kaggle.com). Recall that the OU 
process solves the difference equation 


X(t+ At) = X(t) + 0(u — X(t))At + oVAtB(1). 


Using At = 1, we can rewrite this equation as 


X(t+1)— X(t) = 6-0 X(t) +0 B(I), 


and note that this has the form of a linear regression of X(t + 1) — X(t) 
on X(t). Denoting by a and b the fitted intercept and slope, respectively, we 
can write 6 = —b and fi = a/0 = = —a/b. The volatility o is estimated as the 
sample standard deviation of the error term. The code below estimates the 
parameters of the OU model and plots the observed and simulated trajectories. 


gasprice.data<- read.csv(file="./gaspricedata.csv", 
header=TRUE, sep=",") 
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#estimating parameters 

inc<- gasprice.data$Price[-1]-head(gasprice.data$Price, -1) 
fit<- glm(inc ~ head(gasprice.data$Price,-1)) 

theta.hat<- -fit$coefficients [2] 

mu.hat<- fit$coefficients[1]/theta.hat 

sigma.hat<- sigma(fit) 


#specifying seed 
set .seed (9467108) 


#simulating OU process 
OU<- c() 
OU[1]<- gasprice.data$Price[1] 


for (i in 2:length(gasprice.data$Date) ) 
OULi]<- OU[i-1]+theta.hat*(mu.hat-OU[i-1])+sigma.hat*rnorm(1) 


#plotting trajectories 

plot(as.Date(gasprice.data$Date), gasprice.data$Price, 
type="1", lty=1, lwd=2, col=3, ylim=c(0,8), xlab="Time", 
ylab="Natural gas price", first.panel=grid()) 

lines (as.Date(gasprice.data$Date) , OU, lwd=2, col=4) 
legend("bottomright", c("Actual price", "Simulated price"), 
lty=1, col=3:4) 
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APPLICATION 9.5. Investors are interested in estimating correlation between 
various financial investments (for example, stock prices, stock option prices, 
bond yields, and commodity prices). To illustrate the concept of correlated 
Brownian motions, below we plot IBM stock prices at the closing of the 
stock market between 4/1/2020 and 3/30/2021, and U.S. 10-year treasury 
bond yields for the same time period. The data were downloaded from 
https://www.investing.com. The yields were rescaled by a multiplicative factor 
of 100 to plot comparable values. 


data<- read.csv(file="./stock_bonds.csv", header=TRUE, 
aca") 


time<- as.Date(data$date) 
IBM<- data$stock_price 
bond<- data$bond_yield*100 


#plotting the trajectories 

plot(time, IBM, type="1", lty=1, lwd=2, col="blue", 
ylim=c(0,200), xlab="Time", ylab="Stock price / Bond yield", 
first .panel=grid()) 

lines(time, bond, lwd=2, col="orange") 

legend("bottomright", c("IBM stock", "10-year bond"), lty=1, 
lwd=3, col=c("blue", "orange") ) 
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From the graph, the two curves exhibit somewhat similar behavior. To esti- 
mate the correlation coefficient between these two processes, we model them 
as correlated Brownian motions with the correlation coefficient p. To remove 
the time dependence, we resort to considering fixed-time increments where 
time steps are of size 1. The new processes are still correlated Brownian mo- 
tions with the same correlation coefficient p (see Exercise 9.15 for proof). The 
sample Pearson correlation coefficient, computed on the increments, is the 
maximum-likelihood estimator of p. In our setting, the estimated correlation 
coefficient between IBM stock prices and 10-year treasury bond yields is about 
0.36. The code and output follow. 


#computing increments 
IBM.diff<- IBM[-1]-head(IBM, -1) 
bond. diff<- bond[-1] -head (bond, -1) 


#estimating correlation coefficient 
cor(IBM.diff, bond.diff) 


0.3612484 


Exercises 


EXERCISE 9.1. Let {B(t), t > 0} be a standard Brownian motion. Show that 
the correlation between B(s) and B(t) is 


EXERCISE 9.2. Show that the following processes are standard Brownian mo- 
tions. 

(a) X(t) =tB(1/t) ift > 0, and 0 if t = 0, where B(t) is a standard Brownian 
motion. 

(b) Y(t) = a By (t) + V1 — a? Bo(t), t > 0, where By (t) and Bo(t) are inde- 
pendent standard Brownian motions, and 0 < a < 1. 


EXERCISE 9.3. Let {B(t), t > 0} be a standard Brownian motion. Find the 
probability that 
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(a) 0 < B(1) < land1 < B(3)— B(1) < 3. 
(b) 0 < B(1) < land1 < B(2) < 3. Calculate numeric value in R. 
(c) 0 < B(1) < landO < B(2) < w. 


EXERCISE 9.4. Let {B(t), t > 0} be a standard Brownian motion. Suppose 
0 < s < t. Show that the distribution of B(s) + B(t) is normal with mean 0 
and variance 3s + t. 


EXERCISE 9.5. Let {B(t), t > 0} denote a standard Brownian motion, and 
let M(t) = maxo<s<; B(s). By Proposition 9.5, the cumulative distribution 
function of M(t) is Fur(t)(z) = 20(2/Vt) — 1, x > 0. 

(a) Prove that M(t) has the same distribution as |B(t)|, the absolute value 
of B(t). 

(b) Show that the expected value of M(t) is E(M(t)) = ,/#. 


as 


(c) Compute the mean of the maximum on the interval [0,5]. Simulate 1,000 
trajectories of a standard Brownian motion on this interval and find an em- 
pirical estimate of the mean of the maximum. 


EXERCISE 9.6. Let {B(t), t > 0} denote a standard Brownian motion. Show 
that 

(a) {X(t) =—B(t), t > 0} is also a standard Brownian motion. 

(b) P(mino<s<¢ B(s) < x) = 2(a/Vt) where x < 0. 

(c) Find the probability that the minimum of a standard Brownian motion is 
below -3 on the interval [0, 5]. 

(d) Generate 1,000 trajectories of a standard Brownian motion on the inter- 
val [0,5] and find the sample probability that the minimum falls below -3. 


EXERCISE 9.7. (a) Let {B(t), t > 0} be a standard Brownian motion. Show 
that 


gijae =e BOS eG 
0, ift=1 


is a Brownian bridge. 


(b) Let {X(t), 0 <t< 1} be a Brownian bridge. Show that 


B(t)=(1+ )x(—), #20: 


is a standard Brownian motion. 
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(c) Suppose {X(t),0 < t < 1} is a Brownian bridge and Z is a stan- 
dard normal random variable independent of the Brownian bridge. Show that 
B(t) = X(t) +tZ is a standard Brownian motion on (0, 1]. 


EXERCISE 9.8. Let {B(t), t > 0} be a standard Brownian motion. 
(a) Show that for 0 < s < ¢, the conditional distribution of B(s), given 


B(t), is normal with mean $ B(t) and variance ; (t — s). 


(b) Let B(t) = 0. Argue that the process in part (a) is a Brownian bridge 
on the interval [0,¢]. Note: This gives us another way to define a Brownian 
bridge, as a Brownian motion conditioned on the value at the endpoint. 


EXERCISE 9.9. Let {B(t), 0 <t < T} be a standard Brownian motion on the 
interval [0,7], and let 1/(T’) denote the maximum of this process. 

(a) Use the reflection principle to argue that P(M(T) > a,B(T) < x) = 
P(B(T) > 2a-—2),a>0,4 <a. 

(b) Show that the conditional density of M(T) given that B(T’) = x has the 
form 


2(2a — x _ 2a(a—a) 
fucry|ecr) (lz) = 20), T ,a>0,4<a. 


(c) Denote by Mgr(T) the maximum of a Brownian bridge on the interval 
(0, T]. By the result of Exercise 9.8, the maximum of a Brownian bridge is the 
maximum of a Brownian motion conditioned on the value at time T. Use the 
formula derived in part (b) to show that the density of Mgp(T) is 


4a a2 
fMee(T)(@) = Fe *r ,a>O0. 


1 T 
(d) Show that the expected value of Mgpa(T) is 5 a 


EXERCISE 9.10. A herd of bison graze on a field and return to the water source 
once a day. 

(a) Model the daily movement of the herd as a two-dimensional Brownian 
bridge, where both coordinates are independent Brownian bridges. Use min- 
utes as time units. Plot a simulated trajectory. 

(b) Suppose the home range of the herd is rectangular in shape and the linear 
distance unit is one-tenth of a mile. Find its expected area in square miles. 
Hint: Use the formula for the mean value of the diameter of a one-dimensional 
home range derived in Application 9.1. 
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(c) Simulate 1,000 trajectories of the daily movement of the herd and pro- 
duce an empirical estimate of the area covered, assuming a rectangular shape 
of the home range. How does it compare to the theoretical value from part (b)? 


EXERCISE 9.11. Ornithologists have collected data on bird population size in 
a bird viewing preserve for 5 years (60 months). The data are given in the 
table below. 


These data can be modeled as a Brownian motion with drift and volatility. 
(a) Plot the data. 

(b) Compute the increments and construct a histogram. Are the increments 
normally distributed? 

(c) Estimate the drift and volatility coefficients. 

(d) Simulate a Brownian motion with the estimated parameters. Overlay the 
actual and simulated data on the same plot. 


EXERCISE 9.12. The United States Environmental Protection Agency mon- 
itors an Air Quality Index (AQI) in a certain region. The data given below 
contain the values of AQI for 100 consecutive days for that region. 
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(a) Plot the values of AQI against time (in days). Argue that the data may 
be modeled via a geometric Brownian motion. 

(b) Estimate the parameters of the geometric Brownian motion model. 

(c) Plot the actual and simulated values in the same coordinate system. 


EXERCISE 9.13. The current price of a stock is $150. Suppose that the price 
of the stock changes according to a geometric Brownian motion with the drift 
coefficient 4 = —0.4 and variance o? = 0.76. Use the Black-Scholes-Merton 
option pricing model to calculate the cost of an option to buy the stock at 
time t = 7 for a cost of $120. 


EXERCISE 9.14. Foreign currency exchange rates can be modeled well with 
an Ornstein-Uhlenbeck (OU) process. 

(a) Explain in simple words why the mean-reverting property and bounded 
variance are expected in this setting. 

(b) The data file “Foreign Exchange_Rates.csv” (https://www.kaggle.com/ 
brunotly/foreign-exchange-rates-per-dollar-20002019) contains daily exchange 
rates for some currencies (as a ratio to US dollars) between 1/3/2000 and 
12/31/2019. Select a currency and estimate the parameters of the OU pro- 
cess. Plot actual and simulated trajectories on the same graph. 


182 Brownian Motion 


EXERCISE 9.15. Let {Bi(t),t > 0} and {B2(t),t > 0} be two independent 
standard Brownian motions. Consider a new process {B3(t),t > 0} formed as 
a linear combination of these two processes: B3(t) = p By(t) + V/1— p? Ba(t) 
for some fixed p,-1 <p <1. 

(a) Show that {B3(t),t > 0} is a standard Brownian motion. 

(b) Show that {Bi (t),t > 0} and {B3(t),t > 0} are correlated with the cor- 
relation coefficient p. 

(c) Show that for some fixed s < t, the increments B,(t) — Bi(s) and 
B3(t) — B3(s) are correlated with the correlation coefficient p. 

(d) Download historical data from investing.com website for financial invest- 
ments of your choice, plot the two processes, and estimate the correlation 
coefficient between them. 
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CHAPTER 1 


0.3 0.4 0.3 
EXERCISE 1.1. For a Markov chain with a one-step transition probability matrix o2 0.3 05| 

0.8 0.1 0.1 
we compute: 


(a) P(X; = 2 |X) = 1, X, = 2, X, = 3) = P(X3 = 2|X2.=3) (by the Markov property) 


= Px, = 0.1. 
(b) P(X, = 3 |Xp = 2, X3 = 1) = P(X, =3|X3=1) (by the Markov property) 
=P, = 0.3. 


(c) P(X) =1,X, = 2,X> = 3,X3 = 1) = P(X; =1]X) =1,X, = 2,X> = 3) P(X, = 3X) = 1, 
X, = 2) P(X, = 2|X_ = 1) P(Xp = 1): (by conditioning) 
= P(X; = 1|X, = 3) P(X, = 3 |X, = 2) P(X, = 2| Xp = 1) P(X = 1) (by the Markov property) 
= Ps, Pos Pr» P(X = 1) = (0.8)(0.5)(0.4)(1) = 0.16. 


(d) We first compute the two-step transition probability matrix. We obtain 


0.3 0.4 0.3]/0.3 0.4 0.3 0.41 0.27 0.32 
Pp) = oz 0.3 05 0.2 0.3 05] -| 0.52 0.22 0.26} 
0.8 0.1 0.1110.8 0.1 0.1 0.34 0.36 0.30 


Now we write 
PERG 1G Se HB ke SS 1) = Pe 1 [XG SI a 2 Se Pe HS 3 [Ky SH 
X, = 2) P(X, =2 |X = 1) P(X = 1) $ (by conditioning) 
= P(X, = 1| X;3 = 3) P(X, = 3 | X, = 2) P(X, = 2 | Xp = 1) P(X = 1) (by the Markov property) 
= P?) p® p,, P(X) = 1) = (0.34) (0.26)(0.4)(1) = 0.03536. 


EXERCISE 1.2. (a) We plot a diagram of the Markov chain. 


specifying transition probability matrix 
Em<= matt raise: Gl; 0% 0), O07 “O25; 0, 04. 0; 0 25.< O22',--0:,- 0; 
O, O, 1, 0, O, O, O, O, 1, 0), nrow=5, ncol=5, byrow=TRUE) 


transposing transition probability matrix 
tm.tr<- t(tm) 


plotting diagram 
library (diagram) 
plotmat(tm.tr, arr.length=0.25, arr.width=0.1, box.col="light blue", 
box.lwd=1, box.prop=0.5, box.size=0.12, box.type="circle", cex.txt=0.8, 
lwd=1, self.cex=0.3, self.shiftx=0.01, self.shifty=0.09) 


State 2 is reflective. The chain leaves that state in one step. Therefore, it forms a separate transient 
class that has an infinite period. 


Finally, states 3, 4, and 5 communicate and thus belong to the same class. The chain can return to 
either state in this class in 3, 6, 9, etc. steps, thus the period is equal to 3. Since there is a positive 
probability to leave this class, it is transient. 


The R output supports these findings. 


#creating Markov chain object 
library (markovchain) 
mc<- new("markovchain", transitionMatrix=tm,states=c("1", "2", "3", "4", "5")) 


#computing Markov chain characteristics 
recurrentClasses (mc) 


mq 
transientClasses (mc) 
non 

sguomqmompn 
absorbingStates (mc) 
mq 


(c) Below we simulate three trajectories of the chain that start at a randomly chosen state. 


Hspecifying total number of steps 
nsteps<- 25 


specifying seed 
set.seed (4955145) 
initial 
a2 Oey 


probability 
0.2), 022) 


specifying 
p0<- c(0.2, 


specifying matrix containing states 
MC.states<- matrix(NA, nrow=nsteps, ncol=3) 
simulating states 
for (i in 1:3) { 
state0<- state0<- sample(1:5, 1, prob=p0) 


MC.states[,i]<- rmarkovchain(n=nsteps-l, 
include.t0O=TRUE) 


GI 


} 


#plotting simulated trajectories 
matplot(MC.states, type="1", lty=1, 
xlab="Step", ylab="State", 


lwd=2, 


axis(side=1, at=c(1,5,10,15,20,25)) 


object=mc, t0=state0, 


col=2:4, xaxt="n", ylim=c(1,5), 


panel.first=grid()) 


points(l:nsteps, MC.states[,1], pch=16, col=2) 
points(l:nsteps, MC.states[,2], pch=16, col=3) 
points(l:nsteps, MC.states[,3], pch=16, col=4) 
wo 
vw 
2 
g 
wo 
N 
1 5 10 15 20 25 
Step 


Since state 1 is an absorbing state, sooner or later, the trajectories transition into this state and don’t 


leave it. 


(d) To find the steady-state probabilities, we need to solve the following equations: 


0 0.5 
(111, 12,13, 14,1) = (11,2, M3, 14,1) | 0. 0.8], with the additional condition 
0 
0 


that 7, + 12+ 13+ 1%,+ 15 = 1. 


1, = 1, + 0.572 + 0.273 

T, = 0 

Tl; =1,=1, =0 

1, + 12+ 13+ 14+ 1, =1 
Tl, =1, 12 = 13 = 1, = 15 = O. This solution is expected because state | is an absorbing state, and 
so the chain ends up spending 100% of the time there. Having a unique stationary distribution, it is an 
ergodic Markov chain. 


Written out, the system becomes . It has the degenerate solution 


Using R, we obtain: 


steadyStates (mc) 

12345 

10000 

(e) Here we plot the unconditional probabilities at time n against the time. 


specifying total number of steps 
nsteps<- 70 


specifying matrix containing probabilities 
probs<- matrix(NA, nrow=nsteps, ncol=5) 


computing probabilities 
probs[1,] <- po 

for(n in 2:nsteps) 

probs[n,]<- probs[n-1,]%*%Stm 


#plotting probabilities vs. step by state 
matplot (probs, type="1", lty=1, lwd=2, col=1:5, ylim=c(-0.1, 1.1), 
xlab="Step", ylab="Probability", panel.first=grid() ) 


legend("right", c("State 1", "State 2", "State 3", "State4", "State5"), lty=1, 
lwd=2, col=1:5) 


1.0 


State 1 
State 2 


State 3 
State4 
Stated 


Probability 
00 02 04 O06 08 


Step 


EXERCISE 1.3. (a) We plot a diagram of the Markov chain. 


#Sspecifying transition probability matrix 
tm<- matrix(c(0,1,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0. 
0,0,0,0,0.2,0.4,0.4,0.3,0,0,0.1,0.3,0.1,0.2,0,0,0, 
0,0,0,0.5,0.2,0.2,0.1),nrow=7, ncol=7, byrow=TRUE) 


#transposing transition probability matrix 
tm.tr<- t(tm) 


#plotting diagram 

library (diagram) 

plotmat(tm.tr, arr.length=0.3, arr.width=0.1, arr.pos=0.58, box.col="light blue", 
box.lwd=1, box.prop=0.5, box.size=0.09, box.type="circle", cex.txt=0.8, lwd=1, 
self.cex=0.3, self.shiftx=-0.07, self.shifty=-0.05) 


(b) States 1 and 2 form a class and it is recurrent. The period is 2. Once the chain transitions into this 


class, it never leaves it and will bounce between the two states. 


State 3 is reflecting. The chain leaves this state in one step. This state forms a class of its own. It is a 


transient class and its period is infinite. 


States 4, 5, 6, and 7 communicate and thus form a class. Its period is one because of the loops. 
This class is transient because with positive probability the chain can leave this state and transition 


into the {1, 2} class. 


From R, we obtain: 


#creating Markov chain object 
library (markovchain) 


mc<- new("markovchain", transitionMatrix=tm, 


WoW, bier LD en) 


#computing Markov chain characteristics 
recurrentClasses (mc) 


mam me 
transientClasses (mc) 
mM 

"ar mg" "6" "7 


absorbingStates (mc) 
character (0) 


#creating irreducible Markov chain objects 
tm.ir<- matrix(c(0,1,1,0),nrow=2, ncol=2, 


byrow=TRUE) 
mc.ir<-new("markovchain", transitionMatrix=tm.ir, 


states=c("1", wom, Wo WAM WH Ww 


tates=c("1","2")) 


#finding periods of irreducible Markov chains 
period(mc.ir) 


2 


(c) Below we simulate two trajectories of the chain that start at a randomly selected state. 


specifying total number of steps 
nsteps<- 25 


specifying seed 
set.seed (3339964) 


specifying initial probability 
po<=— eCl/7, 277, 177, If 7, T77, is ty 27) 


specifying matrix containing states 
MC.states<- matrix(NA, nrow=nsteps, ncol=2) 


simulating states 
for (i in 1:2) { 
state0<- sample(1:7, 1, prob=p0) 


MC.states[,i]<- rmarkovchain(n=nsteps-1, object=mc, 


include.t0O=TRUE) 
} 


#plotting simulated trajectories 


matplot(MC.states, type="1", lty=1, lwd=2, col=3:4, 


xlab="Step", ylab="State", panel.first=grid() ) 
axis(side=1, at=c(1,5,10,15,20,25)) 


points(l:nsteps, MC.states[,1], pch=16, col=3) 
points(l:nsteps, MC.states[,2], pch=16, col=4) 


~ 


State 
4 


Step 


15 


ylim=c (1,7), 


20 


Both simulated trajectories transition to the class {1, 2} sooner or later. 


(d) Below we calculate the limiting probabilities. 


t0O=stated, 


xaxt="n" 


25 


InR: 


#finding steady-state distribution 
round (steadyStates (mc) ,digits=4) 

1 234567 
0.50.500000 


There is a single limiting distribution which means that the chain is ergodic. States 1 and 2 absorb the 


chain and then the chain spends 50% of the time in state | and the other 50%, in state 2. 


(e) Here we plot the unconditional probability vectors p, against n. 


specifying total number of steps 
nsteps<- 60 


specifying matrix containing probabilities 
probs<- matrix(NA, nrow=nsteps, ncol=7) 


computing probabilities 
probs[1,] <- po 

for(n in 2:nsteps) 

probs[n,]<- probs[n-1,]%*Stm 


#plotting probabilities vs. step by state 
matplot (probs, type="1", lty=1, lwd=2, col=1:7, ylim=c(-0.05, 0.6), 
xlab="Step", ylab="Probability", panel.first=grid() ) 


legend("right", c("State 1","State 2","State 3","State 4","State 5","State 6", 
"State 7"), lty=1, lwd=2, col=1:7)state 2","state 3","state 4","state 5","state 
6", "state 7"), lty=1, col=1:7) 


0203 04 05 06 


Probability 


State 7 


Oo 0.4 


0 10 20 30 40 50 60 


Step 


For state 1 and 2 the probabilities converge to 0.5, whereas for all the other states, the probabilities 
converge to zero. The curves settle around step 50. 


10 


EXERCISE 1.4. (a) We plot the diagram of the Markov chain. 


#Sspecifying the transition probability matrix 
tm<- matrix(c(0.1,0.2,0.3,0,0.4,0,0.5,0.5,0,0,0,1,0,0,0,0,0,0,0,1,0,0,0,0.6,0.4), 
nrow=5, ncol=5, byrow=TRUE) 


#transposing the transition probability matrix 
tm.tr<- t(tm) 


#plotting the diagram for the Markov chain 

library (diagram) 

plotmat(tm.tr, arr.length=0.3, arr.width=0.1,box.col="light blue", box.lwd=1, 
box.prop=0.5, box.size=0.09, box.type="circle", cex.txt=0.8, lwd=1, self.cex=0.3, 
self.shiftx=-0.07, self.shifty=-0.05) 


InR: 
#creating Markov chain object 
library (markovchain) 


mc<- new("markovchain", transitionMatrix=tm, states=c("1", "2", "3", "4™, ™5")) 


#computing Markov chain characteristics 
recurrentClasses (mc) 


11 


not Ee 
wa" Mp 


transientClasses (mc) 

mq 

absorbingStates (mc) 

character (0) 

#creating irreducible Markov chain objects 


tm.irl<- matrix(c(0,1,0.6,0.4),nrow=2, ncol=2, byrow=TRUE) 
mc.irl<-new("markovchain", transitionMatrix=tm.ir, states=c("4","5")) 


#finding periods of irreducible Markov chains 
period(mc.irl) 


1 


#creating irreducible Markov chain objects 
tm.ir2<- matrix(c(0.5, 0.5, 1, 0),nrow=2, ncol=2, byrow=TRU 
mc.ir2<-new("markovchain", transitionMatrix=tm.ir, states=c 


—~ Fl 
- LS 


#finding periods of irreducible Markov chains 
period(mc.ir2) 


1 


(c) We simulate two trajectories of the Markov chain. 


specifying total number of steps 
nsteps<- 25 


specifying seed 
set.seed (202870) 


specifying matrix containing states 
MC.states<- matrix(NA, nrow=nsteps, ncol=2) 


simulating states 

for (i in 1:2) { 
state0<- sample(1:5, 1, prob=c(1/5, 1/5, 1/5, 1/5, 1/5)) 
MC.states[,i]<- rmarkovchain(n=nsteps-1, object=mc, t0=state0, 
include.t0=TRUE) 

} 


#plotting simulated trajectories 
matplot (MC.states, type="1", lty=1, lwd=2, col=3:4, ylim=c(1,5), xaxt="n" 
xlab="Step", ylab="State", panel.first=grid() ) 


axis(side=1, at=c(1,5,10,15,20,25)) 


points(l:nsteps, MC.states[,1], pch=16, col=3) 
points(l:nsteps, MC.states[,2], pch=16, col=4) 


The trajectories enter either class {2, 3} or {4, 5} and keep bouncing between the two states within 
each class, possibly remaining for a little bit in state 2 or state 5 because of the loops. 
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State 
3 


Step 
(d) In R, we compute the invariant probability measures. 


round (steadyStates (mc) ,digits=4) 


There are two invariant probability measures: (0, 0, 0, 0.375, 0.625) and (0, 0.6667, 0.3333, 0, 0). The 
chain will settle for one of these distributions, depending on what recurrent class it happens to enter 
{2,3} or {4,5}. Neither of these two invariant measures is considered to be the stationary distribution 
because the chain is non-ergodic and the limiting distribution would depend on the initial state of the 
chain. 


(e) We plot the graphs of unconditional probabilities against time, assuming successively that the 
chain starts in states 1, 2,3, 4, and 5. 


We run the following R code five times, each time changing the initial state. 


specifying total number of steps 
nsteps<- 20 


specifying matrix containing probabilities 
probs<- matrix(NA, nrow=nsteps, ncol=5) 


computing probabilities (initial state 1) 

probs[1,] <- c(1,0,0,0,0) 

(state 2) c(0,1,0,0,0) (state 3) c(0,0,1,0,0) (state 4) c(0,0,0,1,0) 
(state 5) c(0,0,0,0,1) 

for(n in 2:nsteps) 

probs[n,]<- probs[n-1,]%*%Stm 


#plotting probabilities vs. step by state 
matplot (probs, main="Initial State 1", type="1", lty=1, lwd=2, col=1:5, 
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ylim=c(-0.05, 1.1), xlab="Step", ylab="Probability", panel.first=grid() ) 


legend("topright", c("State 1", "State 2", "State 3", "State 4", "State 5"), 
lty=1, lwd=2, col=1:5) 
We obtain the following five graphs. 
For the initial state 1: 
> probs 


ree [,2] [,3] [,4] [,5] 
[20,] le-19 0.3703699 0.1851856 0.1666536 0.2777908 


Initial State 1 
State 1 
fas) State 2 
= State 3 
State 4 
co State 5 
oO 
£ © 
= Oo 
oO 
Ee | 
e 7 
ao 
N 
oO 
o 
Oo 


Step 


(2) (0, 0.6667, 0.3333, 0,0) + (=) (0, 0, 0, 0.375, 0.625) = (0, 0.3704, 0.1852, 0.16667, 0.27778). 
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For the initial state 2: 


> probs 
[,1] [,2] [,3] (,4] [,5] 
[20,] 0 0.6666660 0.3333340 0 0 If the chain starts in state 2, it will 


remain within the class {2, 3}, and 

the respective probabilities will 

converge to the invariant vector 
State 1 (0, 0.6667, 0.3333, 0, 0). 


Initial State 2 


1.0 


124 
5 
w 
w 


State 5 


0.8 
1 


Probability 
06 
n 


04 


0.2 
1 


0.0 
1 


For the initial state 3: 
If the chain starts in state 3, it first 


PREODS goes to state 2 with probability one, 
and then will transition within the class 
i] 1 H] 2 H] 3 E] 4 E] 5 . “a0 
at) pen i on con {2, 3}, and the respective probabilities 
[20,] 0 0.6666679 0.3333321 0 0 will converge to the invariant vector 
ieee (0, 0.6667, 0.3333, 0, 0). 
2 ow | 
a T T T T 
2 10 15 20 
Step 
> probs 
Lik [,2] [53] [,4] [,5] 
[20,] 0 0 0 0.3749619 0.6250381 
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Initial State 4 


If the chain starts in state 4, it first 
State 1 


o | —* Shee transitions into state 5 with 
i. —. Ses probability one, and then will stay 


0.8 


=e within the class {4, 5}, and the 
respective probabilities will 
converge to the invariant vector 
(0, 0, 0, 0.375, 0.625). 


Probability 


Step 
For the initial state 5: 
> probs 
ie) [i2] [,3] [,4] [ys] 
[20,] 0 0 0 0.3750229 0.6249771 
Initial State 5 : ; 
reid If the chain starts in state 
== mae 5, it transitions within the 
= 7 — Sate 3 class {4, 5}, and the 
—— State 4 : ca iste 
«| — State 5 respective probabilities 
° will converge to the 
= = 4 invariant vector 
8 (0, 0, 0, 0.375, 0.625). 
a <4 


EXERCISE 1.5. (a) In a box, there are two red (R), four blue (B), and eight green (G) balls. One 
ball is drawn at a time without replacement and its color is noted. The stochastic process 
{X,, n = 1,2,...} with the state space S = {R, B, G} doesn’t satisfy the Markovian property. It can be 


proved, for example, as P(X3 = G |X; = R,X2 = B) = =, whereas P(X3 = G |X, =G,X, =B)= 
=, thus, the color of the ball drawn at the third step depends on the colors of all previously drawn 
balls, not just the one drawn at step two. 
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(b) If the drawing is done with replacement, the process is a Markov chain. Since the balls are put back 
into the box, the colors of drawn balls are independent of each other. Let C stand for any of the three 
colors: red, blue, or green. Then we can write 


P(X, =C |X, = C,X2 = C) = P(X3 = C) = P(X3 = C |X, = C), 


and thus, the Markov property always holds. Note that a sequence of independent trials is a special 
case of a Markov chain. 


EXERCISE 1.6. Let O denote any outcome of a coin flip. The flips are considered independent, 
therefore, we obtain 


P(X3 = O |X; = 0,X, = O) = P(X3 = O) = P(X3 = O |X, = 0), 


that is, the Markovian property always holds. The coin is fair, hence, the transition probability matrix 
is 
H r 
0.5 0.5 
7 10.5 0.5 


05 0.5 


|. To derive the limiting probabilities, we solve (ty,17) = (1y,T17r) | 05 05 


where 1, + 2, = 1. The solution is 1, = m7 = 0.5. Note that a sequence of independent trials is a 
special case of a Markov chain. 


EXERCISE 1.7. (a) Below we find the transition probability matrix for Chapter 1 of “Moby Dick” 
by Herman Melville. 


library (tidyverse) 
chapterl <- read file("./Loomings.txt") 


#Cleaning the text 
lowercase<- tolower(chapter1) 


no.blanks<- gsub(" ","", lowercase) 

no.line.breaks<- gsub("\r\n", "", no.blanks) 
removing all punctuation 

clean.string<- gsub("[{[:punct:]]","",no.line.breaks) 


splitting the string into characters 
x2<- strsplit(clean.string, "") 


shifting the text by one place 

no.last<- substr(clean.string, 1, nchar(clean.string)-1) 
first.blank<- str_c(" ", no.last) 

xl<- strsplit(first.blank,"") 


vowels<-c Can, Nem ea HOM, way) 
consonants<- c ("b", nem, Lie Les Wem Wo", TMs, Wy We we, bi Ls wm"; Ww, Wor, We i ale Woths WwW 
Wah "Ww", Moet Wy, Wry 


for (counter in 1:nchar (x2) 
v<- ifelse(x2[[counter]] %i 
e<x- ifelse(x2[[counter]] %i 


oe AS 


vowels,1,0) 


) 
n 
n% consonants,1,0) 
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vv<- ifelse(xl[[counter]] %in% vowels & x2[[counter]] %in% vowels,1,0) 

ve<- ifelse(xl[[counter]] %in% vowels & x2[[counter]] %in% consonants,1,0) 
ev<- ifelse(xl[[counter]] %in%S consonants & x2[[counter]] %in% vowels,1,0) 
ec<- ifelse(xl[[counter]] %in%S consonants & x2[[counter]] %in% consonants,1,0) 


sum (vv) 
572 

sum (vc) 
3075 


sum (cv) 


3075 
sum (cc) 


2795 


We check quickly that these numbers add up properly. Since the first chapter of “Moby Dick” starts 
and ends with consonants, all vowels are transitioned into and transitioned from. Thus, 

sum (v) =3647=sum (vv) +sum (vc) =572+3075=sum (vv) +sum (cv). Also, all but the last 
consonant are transitioned from, therefore, sum (c) -1=5870=sum(cv) +sum(cc) =3075 
+2795, and all but the first consonant are transitioned to, so sum (c) -1=5870=sum (vc) 

+sum (cc) =3075+2795. The transition probability matrix is 


V Cc 
5? _ 915684 2272 — 0.84316 
3647 3647 
& (22229529395 222 = 0.47615 
5870 5870 


The code below computes the limiting probabilities and the proportions of vowels and consonants in 
the text. 


#Sspecifying the transition probability matrix 
tm<- matrix(c(sum(vv)/sum(v), sum(vc)/sum(v), sum(cv)/(sum(c)-1), 
sum(cc)/(sum(c)-1)), nrow=2, ncol=2, byrow=TRUE 


—_ 


#creating Markov chain object 
library (markovchain) 
mc<- new("markovchain", transitionMatrix=tm, states=c("v","c") ) 


#computing limiting probabilities 
steadyStates (mc) 
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Vv Cc 
0.383209 0.616791 


#computing proportions of vowels and consonants 
print (prop.vowels<- sum(v)/(sum(v)+sum(c) ) ) 


0. 3831687 


print (prop.cons<- sum(c) /(sum(v)+sum(c) ) ) 


0.6168313 


From the output, the limiting probabilities are equal to the actual proportions of vowels and 
consonants. 


(b) Now, we run the same code as in part (a), but with the text of Chapter 2. The code and output are 


library (tidyverse) 
chapter2 <- read file("./The Carpet-Bag.txt") 


#Ccleaning the text 
lowercase<- tolower (chapter2) 


no.blanks<- gsub(" ","", lowercase) 

no.line.breaks<- gsub("\r\n", "", no.blanks) 
removing all punctuation 

clean.string<- gsub("[{[:punct:]]","",no.line.breaks) 


splitting the string into characters 
x2<- strsplit(clean.string, "") 


shifting the text by one place 

no.last<- substr(clean.string, 1, nchar(clean.string) -1) 
first.blank<- str_c(" ", no.last) 

x1l<- strsplit(first.blank,"") 


vowels<-c ("a", HEM i: Ms Urania "Uy") 
consonants<-_ e("b"; W (OLR ELEM LIEK o RL a Lg Re ol a (| LER RAL OLR o RLS iL me SL a ake 
st glia "Ww", he ele he Ale Wgst) 


For ona in 1l:nchar (x2) ) { 


v<- ifelse(x2[[counter]] %in%t vowels,1,0) 

ce<x- ifelse(x2[[counter]] %in%S consonants,1,0) 

vv<- ifelse ( a cee S$inS vowels & x2[[counter]] %in%S vowels,1,0) 

ve<- ifelse(xl[[counter]] tint vowels & x2[[counter]] %in%S consonants,1,0) 
cev<- ifelse(xl[[counter]] Sint consonants & x2[[counter]] %in% vowels,1,0) 
cc<- ifelse(xl[[counter]] %Sin% consonants & x2[[counter]] %in% consonants,1,0) 
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340 
sum (vc) 
1978 
sum (cv) 
1978 


sum (cc) 
1921 


Chapter 2 starts and ends with vowels, all consonants are transitions into and transitioned from. 
Therefore, we must have sum (c) =3899=sum (vc) +sum (cc) =1978+1921=sum (cv) 
+sum(cc). Also, all but the last vowel are transitioned from, so sum (v) -1=2318=sum (vv) 
+sum (vc) =340+1978, and all but the first vowel are transitioned into, so sum (v) -1=2318 
=sum (vv) +sum (cv) =340+1978. The transition probability matrix is 


V Cc 
3* ~ 0914668 —2 = 0.85332 
2318 2318 
ec |@2=050731 ~*=0.49269 
3899 3899 


These transition probabilities are not exactly equal to the ones in Chapter | but are within 1/100th, 
which is very close. 


EXERCISE 1.8. The sentence “The quick brown fox jumped over the lazy dog” repeated 500 times 


is a deterministic sequence of vowels and consonants and cannot be modeled as a Markov chain. To 
prove this mathematically, we do the following calculations. 


thequickbrownfoxjumpedoverthelazydog|thequick... 
CCVCVVCCCCVCCCVCCVCCVEVEVCCCVCVCCCVC|CCVCVVCC... 


Let k = 500 denote the number of repetitions. There are 0 vvv subsequences, k vvc subsequences, 
k cvv subsequences, and 10k cvc subsequences. Thus, we compute 


= = -_ i“ P(vvv) = See _ = = 
P(X, =v |X, =v,X, =v) = SUREO. Oke 0, whereas P(X3 = v |X, =v) = 
P(vvv)+P(cvv) O+k es 


PUY) EP GUD) +PCOVE)FP(CUO) = O+REEPIO‘ = 12 + 0, and So, the Markovian property doesn’t hold. 


EXERCISE 1.9. (a) We show that the genotypes of the direct descendant and the second parent 
follow a Markov chain with the state space S = {(AA, AA), (AA, Aa), (AA, aa), (Aa, AA), (Aa, Aa), 
(Aa, aa), (aa, AA), (aa, Aa), (aa, aa)} and the transition probability matrix 
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(AA, AA) (AA, Aa) (AA, aa) (Aa, AA) (Aa, Aa) (Aa, aa) (aa, AA) (aa, Aa) (aa, aa) 

(AA, AA 1/3 1/3 1/3 0 0 0 0 0 0. 
(AA, Aa [ 1/6 1/6 1/6 1/6 1/6 1/6 0 0 o| 
(AA, aa) 0 0 0 1/3 1/3 1/3 0 0 0 
(Aa, AA) 1/6 1/6 1/6 1/6 1/6 1/6 0 0 0 
(Aa, Aa 1/12 1/12 1/12 1/6 1/6 1/6 1/12 1/12 1/12 
(Aa,aa 0 0 ia) 1/6 1/6 1/6 1/6 1/6 1/6 

(aa, AA 0 0 0 1/3 1/3 1/3 0 0 0 
(aa, Aa 0 0 0 1/6 1/6 1/6 1/6 1/6 1/6 

(aa,aa 0 0 0 0 0 0 1/3 1/3 1/3 


e If both parents have the combination AA of genes, their direct descendant will have genes AA with 
probability one, and will equally likely choose the second parent with genes AA, Aa, or aa. 
Therefore, the state (AA, AA) transitions into states (AA, AA), (AA, Aa), and (AA, aa) with 
probabilities equal to 1/3. 

e Ifthe parents have genes (AA, Aa), their direct descendant will have genes AA with probability “% 
or genes Aa with probability 2. And will choose the second parent with either of the three types 
with probability 1/3. Thus, (AA, Aa) transitions into states (AA, AA), (AA, Aa), (AA, aa), (Aa, AA), 


(Aa, Aa), or (Aa, aa), with probability (5) (5) a ; each. 
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(AA, aa), (aa, AA), (aa, Aa), or (aa, aa) with probability (=) (=) a = each, and states 
(Aa, AA), (Aa, Aa), or (Aa, aa) with probability (5) (=) = : each. 


The other cases are proven similarly. 


(b) Below we determine the transient and recurrent classes of the Markov chain. 


#Specifying transition probability matrix 
tm<- matrix(c(1/3, 1/3, 1/3, 0, 0, 0, O, O, O, 1/6, 1/6, 1/6, 1/6, 1/6, 1/6, 0, O, 
0, 0, O, 0, 1/3, 1/3, 1/3, 0, 0, 0, 1/6, 1/6, 1/6, 1/6, 1/6, 1/6, 0, O, O, 

1/12. 1/22, 1/12, 1/6, 1/6, 1/6, 1/12, 1/12, 1/12; 0, Oy ©, 1/6, 1/6) 1/6,-1/6, 
1/6, 1/6, 0, O, 0, 1/3, 1/3, 1/3, 0, 0, O, O, O, O, 1/6, 1/6, 1/6, 1/6, 1/6, 1/6, 
0, O, O, O, O, 0, 1/3, 1/3, 1/3), nrow=9, ncol=9, byrow=TRUE) 


#creating Markov chain object 

library (markovchain) 

mc<- new("markovchain", transitionMatrix=tm, states=c("AAAA", "AAAa", “AAaa", 
"AaAA", "AaAa", "Aaaa", "aaAA", "aaAa", "aaaa") ) 


#computing Markov chain characteristics 
transientStates (mc) 


character (0) 
recurrentClasses (mc) 


"AAAA" "AAAa" "AAaa" "AaAA" "AaAa" "“Aaaa" "aaAA" "aaAa" "aaaa" 
All states are recurrent and none are transient. 


(c) Next, we find the stationary distribution. 
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#finding stationary distribution 
steadyStates (mc) 


AAAA AAAa AAaa AaAA 
0.08333333 0.08333333 0.08333333 0.16666667 
AaAa Aaaa aaAA aaAa 
0.16666667 0.16666667 0.08333333 0.08333333 
aaaa 
0.08333333 
In a long run, there will be about 8.3% of each gene type (AA, AA), (AA, Aa), (AA, aa), (aa, AA), 


(aa, Aa), and (aa, aa), and about 16.7% of each gene type (Aa, AA), (Aa, Aa), and (Aa, aa). 


(d) Here we find the initial state that achieves the stationary distribution in the smallest number of 
generations. To answer this question, we run the following R code nine times with the initial state 
ranging from (1, 0,0,...,0) to (0,0, ...,0,1). We look for the smallest generation number for which the 
steady-state distribution has probabilities that round to 0.0833 and 0.1667. 


specifying total number of steps 
nsteps<- 20 


specifying matrix containing probabilities 
probs<- matrix(NA, nrow=nsteps, ncol=9) 


computing probabilities 
probs[1,] <- c(1, 0, 0, 0, 0, O, 0, O, QO) 


for(n in 2:nsteps) { 
print (n) 
print (probs[n,]<- round(probs[n-1,]%*%Stm, 6) ) 


15 
[,1] [,2] [,3] [,4] [9] [,6] [,7] [,8] [,9] 
[1,] 0.083353 0.083353 0.083353 0.166665 0.166665 0.166665 0.083312 0.083312 0.083312 


16 
9 


[,1] [,2] [,3] [,4] [5] [,6] [7] [,8] [,9] 
[1,] 0.083343 0.083343 0.083343 0.166665 0.166665 0.166665 0.083322 0.083322 0.083322 


The initial state (AA, AA) achieves the steady-state probabilities in the 16th generation. We repeat this 
code for the other 8 initial states and get the following result: 


Initial State 


(AA, AA) 


(AA, Aa) 


(AA, aa) 


(Aa, AA) 


(Aa, Aa) 


(Aa, aa) 


(aa, AA) 


(aa, Aa) 


(aa, aa) 


Smallest 
Generation 

To Achieve the 
Steady-State 
Distribution 


16 


ie) 


15 


|) 


15 


16 


The conclusion is that the most genetically diversified type (Aa, Aa) in the first generation results in 
the limiting distribution already in the second generation, whereas the least diversified genetic types 
(AA, AA) and (aa, aa) need to wait until the 16th generation to see convergence. 
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EXERCISE 1.10. The code below selects the weather conditions for Detroit, and computes the 
empirical conditional probability of clouds tomorrow, given clouds today and yesterday, and the 
conditional probability of clouds tomorrow, given clouds today. 


weather.data<- read.csv("./weather description.csv", header=TRUE 


D 


ET<- weather.dataSDetroit 


table (DET) 


DET 
broken clouds 
3975 
fog 
585 
heavy intensity drizzle 
heavy snow 
249 
rain 
193 
snow 
172 
rain 
1450 
proximity thunderstorm 
104 
snow 
4 
snow 
222 
rain 
14 
rain 
4 


light intensity shower 
light shower 


moderate 


shower 


thunderstorm with heavy 


very heavy 


X3<- ifelse(DET=="Sky is c 
"few clouds", 


"overcast clouds", 
ifelse (DET %in%s c("heavy shower snow", 


, sep=", ") 


drizzle few clouds 
96 1775 
freezing rain haze 
3 768 
heavy intensity rain heavy shower snow 
437 107 
light intensity drizzle light intensity drizzle rain 
396 2 
light rain light shower sleet 
3873 1 
light snow mist 
1383 3414 
overcast clouds proximity shower rain 
6470 208 
scattered clouds shower rain 
3940 7 
sky is clear smoke 

15249 
squalls thunderstorm 

1 
thunderstorm with light rain thunderstorm with rain 
41 34 

lear", "no clouds", ifelse(DET %in% c("broken clouds", 


"scattered clouds", 
"heavy snow", 


snow", "shower snow", "snow"), "snow", "rain"))) 
table (X3) 
x3 
clouds’ no clouds skyclear snow 
16161 11705 15249 2137 
library (Hmisc) 
X2<- Lag (X3,shift=1) 
X1<- Lag (X3,shift=2) 
library (Hmisc) 
X2<- Lag (X3,shift=1) 
X1<- Lag (X3,shift=2) 
#computing P(X3=c|X2=c,X1=c) 
ccec<- ifelse (X1=="clouds" & X2=="clouds" & 
cen<- ifelse (X1=="clouds" & X2=="clouds" & 
cer<- ifelse (X1=="clouds" & K2=="clouds" & 
cces<- ifelse(X1=="clouds" & X2=="clouds" & 
( 


sum(ccc) /sum 


#computing P(X3=c|X2=c) 


nec<- ifelse(X1=="no clouds" 
& X2=="clouds" 


rece<- ifelse (X1=="rain" 


ccct+tcecen+ccrt+ccs) 


& X2=="clouds" 


"smoke"), "clouds", 
"light shower snow", 


X3=="clouds",1,0) 
X3=="no clouds",1,0) 
X3=="rain",1,0) 
X3=="snow",1,0) 


& X3=="clouds",1,0) 
& X3=="clouds",1,0) 


"light 


scc<- ifelse 
nen<- ifelse 
ner<- ifelse 
nces<- ifelse 


1=="snow" & X2=="clouds" & X3=="clouds",1,0) 

1=="no clouds" & X2=="clouds" & X3=="no clouds",1,0) 
X1=="no clouds" & X2=="clouds" & X3=="rain",1,0) 
X1=="no clouds" & X2=="clouds" & X3=="snow",1,0) 


Xx 
Xx 


ren<x- ifelse(X1l=="rain" & X2=="clouds" & X3=="no clouds",1,0) 
X1l=="rain" X2=="clouds" X3=="rain",1,0) 
res<- ifelse (X1=="rain" X2=="clouds" X3=="snow",1,0) 


scn<- ifelse 
scr<- ifelse 


( 
( 
( 
( 
( 
rer<- ifelse ( 
( 
( 
( 
scs<- ifelse ( 


& & 
& & 
X1=="sSnow" & X2=="clouds" & X3=="no clouds",1,0) 
X1=="snow" & X2=="clouds" & X3=="rain",1,0) 
X1=="sSnow" & X2=="clouds" & X3=="Snow",1,0) 


sum(ccct+nectrectscc) /sum(ccc+ecntccr+ccstncectnent+nertnes+rectren 
trer+rcestscctscnt+scr+scs) 


The state space of this process is S = {c = "clouds”,n = "no clouds”,r = "rain’”,s = ”snow”}. 
From the above output, 
P(X = G55 = 6,45 =) 
P(X, =c|X, =c,X> =c) = ——_——___——— 
(X3 =c|X, =c,X2 =c) P(X, =c,X,=c) 
= zee) = 0.8143572 
~ P(ccc) + P(cen) + P(ccr)+P(ccs) : 


whereas 


P(X3 =c|X,=c)= EA EE Es 0.7662892 
oleae ©) ~ P(cec) + P(cen) + P(cer) +++ P(scs) 


Since the two quantities are not the same, the process is not a Markov chain. 


EXERCISE 1.11. Let s denote the air quality status (good/unhealthy/hazardous). We are given that 
P(Xna1 = S| Xn = 8, Xn-1 = Se) Xy = S) = P(Xngy = 5 | Xn = 5,Xn_-1 = 5). We consider states as 
air quality statuses in two consecutive days: (X,, Xz), (Xz, X3), etc. We show that the Markov property 
holds: 


P((Xn Xnsi) = (8,8) | Xn Xn) = (5,5), Xr, X2) = (8,5) 
=P Keay 2 Bh HS Aq = She = Syed = 8) 

— PXns1 = 5,Xn = 5, Xn-1 = S$, X1 = 5) 

7 PO So So tS 

=P Xig9. = 8| X54 = Si XgHa = Se = 8) 

= P(Xn41 = S| Xn = 5,Xn-1 = 8) (by the assumption of the problem) 
= P(Xna1 = 5,Xn = 8,Xn-1 = 5) 
P(Xp = 8,Xp-1 = S) 
= P((Xn,Xn41) = (S,5)|(Xn-1 Xn) = (5,5). 


EXERCISE 1.12. (a) Let state 1 give income $200, state 2 give income $0, state 3 give income - 
$75, state 4 give income $105, and state 5 give income -$130. Since the rolls of the die are 
independent, the next state will depend only on the present state. We use the fact that the die is fair and 
that the states are traversed circularly, and write the transition probability matrix: 
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(b) We compute the steady-state probability of each square and find the long-run winning of the 
player. 


#specifying transition probability matrix 
tm<- matrix(c(1/6, 1/3, 1/6, 1/6, 1/6, 
1/6, 1/6, 2/3, 1/6, 1/6; 
1/6, 1/6, 1/6, 1/3, 1/6, 
1/6, 1/6, 1/6, 1/6, 1/3, 
1/3, 1/6, 1/6, 1/6, 1/6), 

nrow=5, ncol=5, byrow=TRUE) 


#creating Markov chain object 
library (markovchain) 
mc<- new("markovchain", transitionMatrix=tm, states=c("1", "2", "3", "4", "5")) 


steadyStates (mc) 
Le 72 3 4 5 
0.2 0.2 0.2 0.2 0.2 


In the long-run, the chain will be uniformly distributed between the five states, and thus the expected 
winning of the player will be E(winning) = ($200)(0.2) + ($0)(0.2) + (—$75)(0.2) + 
($105) (0.2) + (—$130) (0.2) =$20. 


EXERCISE 1.13. (a) Assuming that the traffic starts with the light state at 1PM, we find the 
distribution of the states at 6PM. Traffic conditions change every 20 minutes, therefore between 1PM 
and 4PM there will be 9 transitions between the states (light/heavy/jammed), and between 4PM and 
6PM there will be 6 transitions. We run the following R code to find the distribution of states at 6PM: 


specifying the transition probability matrices 

tml<- matrix(c(0.4, 0.4, 0.2, 0.3, 0.5, 0.2, 0, 0.5, 0.5), nrow=3, ncol=3, 
byrow=TRUE) 
tm2<- matrix(c(0.1, 0.5, 0.4, 0.1, 0.3, 0.6, O, 0.1, 0.9), nrow=3, ncol=3, 
byrow=TRUE) 


computing the unconditional distribution at 6pm 
library (expm) 
statelpm<- c(1, 0, 0) 


fo) 


state4pm<- statelpms*% (tm1%% 
print (state6pm<- state4pms*% 


[,1] bed Ee 
0.01504877 0.1332313 0.85172 


9) 
tm2%*%S6) ) 


% 
( 


P(light traffic) =0.01504877, P(heavy traffic) = 0.1332313, and P(jammed traffic) = 0.85172. 
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(b) Below we simulate 10,000 trajectories to verify the result of part (a). 


#creating Markov chain objects 

library (markovchain) 

mcl<- new("markovchain", transitionMatrix=tml, states=c("light", "heavy", 
"jJammed") ) 


mc2<- new("markovchain", transitionMatrix=tm2, states=c("light", "heavy", 
"jJammed") ) 


#simulating states between lpm and 4pm 
MC.states4pm<- matrix(NA, nrow=9, ncol=10000) 


for (i in 1:10000) 
MC.states4pm[,i]<- rmarkovchain(n=9, object=mcl, t0="light") 


#Simulating states between 4pm and 6pm 
MC.states6pm<- matrix(NA, nrow=6, ncol=10000) 


for (i in 1:10000) 
MC.states6pm[,i]<- rmarkovchain(n=6, object=mc2, t0=MC.states4pm[9,i]) 


#concatenating two matrices 
MC.states<- rbind(MC.states4pm, MC.states6pm) 


#computing frequencies of states at 6pm 
table (MC.states[15,]) 


heavy jammed light 
1316 88533 151 


Thus, the estimates are P(light traffic) =0.0151, P(heavy traffic) = 0.1316, and 
P(jammed traffic) = 0.8533. 


EXERCISE 1.14. (a) Assuming that a shrub is initially sustainable, we simulate three trajectories of 
the Markov chain. 


#specifying transition probability matrix 
tEm<— matr_x:.(e (0.0; O12; 10.2490. dee On 150. Fy 01 090 Lp 063, O44 > O82), Oy 
O, O, 1), nrow=4, ncol=4, byrow=TRUE) 


Gl 


library (markovchain) 
mc<- new("markovchain", transitionMatrix=tm, states=c("1", "2", "3", "4™")) 


specifying total number of steps 
nsteps<- 25 


specifying seed 
set.seed (912332) 


specifying matrix containing states 
MC.states<- matrix(NA, nrow=nsteps, ncol=3) 
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#simulating states 


for (2 an 23) 4 
state0<- 1 
MC.states[,i]<- rmarkovchain(n=nsteps-1, object=mc, t0=state0, 
include.t0=TRUE) 

} 


#plotting simulated trajectories 
matplot (MC.states, type="1", lty=1, lwd=2, col=2:4, xaxt="n", yaxt="n", 
ylim=c (1,4), xlab="Step", ylab="State", panel.first=grid() ) 


axis(side=1, at=c(1,5,10,15,20,25)) 
axis (side=2, at=c(1,2,3,4)) 


points(l:nsteps, MC.states[,1], pch=16, col=2) 
points(l:nsteps, MC.states[,2], pch=16, col=3) 
points(l:nsteps, MC.states[,3], pch=16, col=4) 


State 


Step 
(b) Here we find the probability that an initially sustainable shrub will eventually become extinct. 
steadyStates (mc) 
1234 
O0O01 
State 4 (“extinct’’) is an absorbing state, thus, eventually, a sustainable shrub will become extinct with 
probability one. 


EXERCISE 1.15. (a) The number of music instruments on Tuesday morning can assume values 3, 4, 
5, 6, or 7. Therefore, the state space consists of these five states. 


Let N ~ Poisson(A) be the number of instruments bought during the week. Then 
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P(3 instruments this week 
P(4 instruments this week 
P(5 instruments this week 
P(6 instruments this week 
P(7 instruments this week 


P(3 instruments this week 
P(4 instruments this week 
P(5 instruments this week 
P(6 instruments this week 
P(7 instruments this week 
= 0.9084 


3 instruments previous week) = P(N = 0) = exp(—4) = 0.0183 

3 instruments previous week) = 0 

3 instruments previous week) = 0 

3 instruments previous week) = 0 

3 instruments previous week) = P(N = 1) = 1 — 0.0183 = 0.9817 


4 instruments previous week) = P(N = 1) = 4 exp(—4) = 0.0733 
4 instruments previous week) = P(N = 0) = exp(—4) = 0.0183 
4 insturments previous week) = 0 

4 instruments previous week) = 0 

4 instruments previous week) = P(N = 2) = 1 — 0.0733 — 0.0183 


2 
P(3 instruments this week | 5 instruments previous week) = P(N = 2) = —exp(—4) = 0.1465 
P(4 instruments this week | 5 instruments previous week) = P(N = 1) = 4 exp(—4) = 0.0733 
P(5 instruments this week | 5 instruments previous week) = P(N = 0) = exp(—4) = 0.0183 
P(6 instruments this week | 5 instruments previous week) = 0 
P(7 instruments this week | 5 instuments previous week) = P(N = 3) = 1 — 0.1465 — 0.0733 


—0.0183 = 0.7619 


P(3 instruments this week 


P(4 instruments this week 
P(5 instruments this week 
P(6 instruments this week 
P(7 instruments this week 


3 
6 instruments previous week) = P(N = 3) = —exp(—4) = 0.1954 


2 
6 instruments previous week) = P(N = 2) = —exp(—4) = 0.1465 
6 instruments previous week) = P(N = 1) = 4exp(—4) = 0.0733 
6 instruments previous week) = P(N = 0) = exp(—4) = 0.0183 
6 instuments previous week) = P(N = 4) = 1 — 0.1954 — 0.1465 


—0.0733 — 0.0183 = 0.5665 


P(3 instruments this week 
P(4 instruments this week 


P(5 instruments this week 


P(6 instruments this week 
P(7 instruments this week 


4 

7 instruments previous week) = P(N = 4) = ~exp(—4) = 0.1954 
3 

7 instruments previous week) = P(N = 3) = —exp(—4) = 0.1954 


2 
7 instruments previous week) = P(N = 2) = —exp(—4) = 0.1465 
7 instruments previous week) = P(N = 1) = 4exp(—4) = 0.0733 
7 instruments previous week) = P(N = 0) + P(N = 5) = 


1 — 0.1954 — 0.1954 — 0.1465 — 0.0733 = 0.5665 = 0.3894 


The one-step transition probability matrix is 


3 4 
0 
0.0183 


0.0733 
0.1465 


NIDA BW 


0.1954 


0.0183 
0.0733 
0.1465 


5 6 7 
0 0 0.9817 
0 0 0.9084 

0 0.7619 


0.5665 
0.3894 


(b) The following code generates inventory trajectories, assuming that the initial inventory size is 
randomly chosen. 


#specifying transition probability matrix 

tm<- matrix(c(0.0183, 0, 0, 0, 0.9817, 0.0733, 0.0183, 0, 0, 0.9084, 0.1465, 
0.0733, 0.0183, 0, 0.7619, 0.1954, 0.1465, 0.0733, 0.0183, 0.5665, 0.1954, 
0.1954, 0.1465, 0.0733, 0.3894), nrow=5, ncol=5, byrow=TRUE) 


GI 


library (markovchain) 
mc<- new("markovchain", transitionMatrix=tm, states=c("3", "4", "5", "6", "7")) 


#specifying total number of steps 
nsteps<- 25 


#specifying seed 
set.seed (8596943) 


#Sspecifying matrix containing states 
MC.states<- matrix(NA, nrow=nsteps, ncol=2) 


#simulating states 


for (i in 1:2) { 
state0<- sample(3:7, 1, prob=c(0.2,0.2,0.2,0.2,0.2) ) 
MC.states[,i]<- rmarkovchain(n=nsteps-1, object=mc, t0=state0, 
include.t0=TRUE) 
} 


r 


#plotting simulated trajectories 
matplot (MC.states, type="1", lty=1, lwd=2, col=3:4, axat="n", ylim=c(3,7), 
xlab="Tuesday morning", ylab="Number of instruments", panel.first=grid() ) 


axis(side=1, at=c(1,5,10,15,20,25)) 


points(l:nsteps, MC.states[,1], pch=16, col=3) 
points(l:nsteps, MC.states[,2], pch=16, col=4 
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(c) Using the definition of conditional probability and Markov property, we can write 


PX = 1X3 = 1X0 = 7; X, =7) 
P(X, = 7,X3 =7,X2 =7| X, = 7) = OO 
(X4 3 2 | 1 ) P(X, =7) 
_ P(X, = 7| X3 = 7,X2 =7, X, = 7) P(X3 = 7,X2 =7, X,=7) 
7 POG SF) 
_ P(X, = 7| X3 = 7) P(X3 = 7|X2 = 7, X1 = 7) P(X =7, X1 =7) 
7 P(X, = 7) 
_ P(Xq = 7 Xa = 7) P(Xg = 7 | Xp = 7) P(X = 7X1 = 7) PX =7) 
P(X, =7) 


= P,7P77P,, = P2, = (0.3894)? = 0.059. 
(d) Below we find the steady-state probability distribution for this Markov chain. 
steadyStates (mc) 
3 4 5 6 7 
0.1487365 0.1300719 0.09080707 0.04379828 0.5865863 
Using these values, we compute the expected weekly storage cost: 
E(weekly storage cost) 


= $5 ((3)(0.1487365) + (4)(0.1300719) + (5)(0.09080707) 
+ (6)(0.04379828) + (7)(0.5865863)) = ($5) (5.789426) = $28.94713. 


Thus, in the long run, there will be, on average, 5.789426 instruments in the store on Tuesday 
morning, and the average storage cost will amount to $28.95. 
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CHAPTER 2 


EXERCISE 2.1. In theory, E(X50) = ((2)(0.3) — 1)(50) = —20 and Var(Xso) = 
(4)(0.3)(1 — 0.3)(50) = 42. 


Next, we run an R code that simulates 10,000 trajectories of length 50 steps and computes the mean 
and variance of the last values. 


#specifying parameters 
b<=.0:.3 

n<- 50 

ntraj<- 10000 


setting seed number 
set.seed (546675) 


defining walk as matrix 
walk<- matrix(NA, nrow=n, ncol=ntraj) 


Simulating trajectories 
tor: (jan slentrasy): -{ 
walk[1,j]<- 0 
for (k in 2:n) { 
walk[k,j]<- ifelse(runif(1)<p, walk[k-1,j]+1, walk[k-1,j]-1) 
} 
} 


mean (walk[50,]) 
-19.5824 


var (walk[50,]) 


42.16583 


The empirical values are pretty close to the theoretical ones. 


EXERCISE 2.2. (a) The R script below simulates 10,000 trajectories and counts how many of them 
have a value of 0 at the 1,000th step. 


setting counter to zero 
nzeros<- 0 


specifying seed 
set.seed (675572) 


defining walk as matrix 
walk<- c() 


simulating trajectories 
for (j in 1:10000) 

{ 

walk[1]<- 0 

for (i in 2:1001) 
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walk[i]<- ifelse(runif (1)<0.5, walk[i-1]+1, walk[i-1]-1) 
if (walk[{1001]==0) nzeros=nzerostl 


} 
print (nzeros) 
253 


(a) The theoretical probability of returning to 0 on the 1,000th step is 


1000 
co) (5) = 0.025. This quantity was computed in R: 


P(Xt000 = 01 Xo = 0) = (Cann 2 


choose (1000,500) *0.5%*1000 


0.02522502 


The estimated probability from part (a) is P(X1999 = 0 | Xp = 0) = — = 0.0253, which is 


a pretty accurate estimate of the theoretical value. 


EXERCISE 2.3. (a) The code below simulates the 10,000 trajectories of one-, two-, and three-dime 
nsional symmetric random walks that start at the origin and continue for at most 1,000 steps. A trajecto 
ry that reaches the origin is terminated. 


#setting counters to zero 


nlD<- 0 
n2D<- 0 
n3D<- 0 


#specifying seed 
set.seed (300799) 


#defining 1D walk as vector 
walk1D<- c() 
nsteps1lD<- c() 


#Simulating 1D trajectories 
for (j in 1:10000) 
{ 
walk1D[1]<- 0 #setting initial value to zero 
for (1 in 2:1001) 
{ 
walk1D[i]<- ifelse(runif (1)<0.5, walk1D[i-1]+1, walk1D[i-1]-1) 
if (walk1iD[i]==0) { 
nilD=n1D+1 
break } 
} 
nsteps1lD[j]=i 


#defining 2D walk as matrix 
walk2D<- matrix(NA, nrow=1001, ncol=2) 
nsteps2D<- c() 


#defining random steps 
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rstep2D<- matrix(c(1, 0, -1, 0, 0, 1, 0, -1), nrow=4, ncol=2, byrow=TRUE) 


#simulating 2D trajectories 
for (j in 1:10000) 
{ 
walk2D[1,]<- c(0,0) #Ssetting initial value to the origin 
for “(ay ain: 28.10.02) 
{ 
walk2D[i,]<- walk2D[i-1,]+rstep2D[sample(1:4, size=1),] 


if (walk2D[i,1]==0 & walk2D[i,2]==0) { 
n2D=n2D+1 
break } 
} 
nsteps2D[j]=i 
} 


#defining 3D walk as matrix 
walk3D<- matrix(NA, nrow=1001, ncol=3) 
nsteps3D<- c() 


#defining random steps 
rstep3D<= matrix (ce(1l; 0; .0,=1, 0, 0, 0, ° 1, 0; 0, —L;, 0,. 0,0; 2; °0,-"0, =-1), 
nrow=6, ncol=3, byrow=TRUE) 


#simulating 3D trajectories 

for (j in 1:10000) 

{ 
walk3D[1,]<- c(0,0,0) #setting initial value to the origin 
OK. “Cis an! -24:0'01)). 

{ 


walk3D[i,]<- walk3D[i-1,]+rstep3D[sample(1:6, size=1),] 
if (walk3D[i,1]==0 & walk3D[i,2]==0 & walk3D[i,3]==0) { 
n3D=n3D+1 
break } 
} 
nsteps3D[j]=i 
} 
print (n1D) 
9756 
print (n2D) 
6759 
print (n3D) 
3329 


Roughly 97.6% of the 1D trajectories returned to 0, about 67.6% of the 2D trajectories returned to 
(0, 0), and only 33.3% of the 3D trajectories returned to (0, 0, 0). 


(a) The average number of steps it took those trajectories to return to the origin is computed as 
mean (nsteps1lD[nsteps1D!=1001]) 
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27.47499 
mean (nsteps2D[nsteps2D!=1001])) 
61.47625 


mean (nsteps3D[nsteps3D!=1001]) 


27 .83689 


The 97.6% of the 1D trajectories that returned to the origin, did it in 27.47 steps, on average. 
The 67.6% of the 2D trajectories that returned to the origin, did it in 61.48 steps, on average. 
The 33.3% of the 3D trajectories that returned to the origin, did it in 27.84 steps, on average. 


EXERCISE 2.4. (a) The R script below simulates the trajectories and terminates them if the barrier 
is hit. Otherwise, trajectories continue for 1,000 steps. The total number of trajectories that hit the 
barrier is counted. We also record the number of steps (for part (b)) and the y-coordinate (for part (c)). 


setting counter to zero 
nhits<- 0 


specifying seed 
set.seed (50118) 


defining walk as matrix 

walk<- matrix(NA, nrow=1001, ncol=2) 
nsteps<- c() 

ycoord<- c() 


#defining random steps 
rstep<- matrix(c(1, 0, -1, 0, 0, 1, 0, -1), nrow=4, ncol=2, byrow=TRUE 


— 


#Simulating trajectories 
for (j in 1:10000) 
{ 
walk[1,]<- c(0,0) #Ssetting initial value to the origin 
for (i in 2:1001) 
{ 
walk[i,]<- walk[i-1,] + rstep[sample(1:4, size=1),] 


if (walk[i,1]==30) { 
nhits=nhitstl 
break } 
} 
nsteps[j]<- i 
ycoord[j]<- ifelse(i==1001, 99999, walk[i,2]) 
} 


print (nhits) 


1764 


So, of the 10,000 trajectories, 1,764 hit the vertical barrier. Thus, the estimated probability to hit the 
barrier is 0.1764. 
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(b) The average number of steps it takes a trajectory to hit the barrier, provided it did hit 
the barrier within the 1,000 steps, is estimated as 


mean (nsteps[nsteps!=1001]) 


623.2053 


It took on average 623.2 steps to hit the barrier for the 17.64% of the trajectories that terminated at 
the barrier. 


(a) Estimate the expected value of the y-coordinate at the time when the random walk hits the barrier. 
What should this value be from the theoretical point of view? Hint: deduce from a symmetry 
argument. 


mean (ycoord[ycoord!=99999]}) ) 
0.1066364 


The estimated average y-coordinate for 17.64% of the trajectories that hit the barrier was 0.1066. 
From the theoretical viewpoint, using the symmetry of the random walk, we can argue that the y- 
coordinate should be equal to 0. 


EXERCISE 2.5. By running the following script, we simulate trajectories and calculate the number 
of those that hit the barrier. The plot is given below. 


Nhits<- c() 


specifying seed 
set.seed (96770) 


defining walk as matrix 
walk<- matrix(NA, nrow=1001, ncol=2) 


defining random steps 
rstep<- matrix(c(1, 0, -1, 0, 0, 1, 0, -1), nrow=4, ncol=2, byrow=TRUE 


— 


varying the barrier value 
for (barrier in 1:50) { 
nhits<- 0 
simulating trajectories 
for (j in 1:100) 
{ 
walk[1,]<- c(0,0) #setting initial value to the origin 
for (i in 2:1001) 
{ 
walk[i,]<- walk[i-1,] + rstep[sample(1:4, size=1),] 


if (walk[i,1]==barrier) { 
nhits=nhitstl 
break } 


} 
} 
Nhits[barrier]=nhits 


} 
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print (Nhits) 


[1] 93 94 93 94 87 85 72 66 66 66 61 61 
[13] 66 43 49 49 44 46 37 38 39 31 33 29 
[25] 30 28 19 24 17 18 23 13 22 12 8 9 
[37] 810 811 6 8 4 7 4 6 3 4 
[49] 4 1 


plot(1:50, Nhits/100, col="blue", xlab="Position of barrier", ylab="Probability of 
hitting barrier", panel.first=grid() ) 
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Position of barrier 


We see that as the barrier value increases from 1 to 50, the estimated probability of hitting this barrier 
decreases from 0.93 to 0.01, in a slightly curvilinear (convex downward) manner. 


EXERCISE 2.6. The lines of code given below terminate each trajectory if it reaches a side of the 
square. The total number of steps required is recorded for each trajectory. The average value is 
computed at the end. 


walk<- data.frame() 
nsteps<- c() 


set.seed (37440) 


#defining random steps 
rstep<- matrix(c(1, 0, -1, 0, 0, 1, 0, -1), nrow=4, ncol=2, byrow=TRUE 


—_ 


#Simulating trajectories 
for (j in 1:1000) 
{ 


walk[1,1]<- 0 
walk[1,2]<- 0 
i<- 2 


repeat { 
walk[i,]<- walk[i-1,] + rstep[sample(1:4, size=1),] 
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if (walk[{i,1]==10 | walk[i,1]==-10 | walk[i,2]==10 | walk[i,2]==-10) 
{ break  } 
else i=itl 


} 


nsteps[j]=1 
} 


mean (nsteps) 


120.898 


Hence, the average number of steps it takes for the random walk to reach the square is estimated as 
120.898 steps. 


EXERCISE 2.7. (a) Conditioning on the outcome of the first step, we see that the probability P; 
solves the recurrence relation P; = pP;,, + qP;_, with the border constraints P, = 0 and Pz = 1. 


i 
Assuming first that : # 1, we look for the solution in the form P; = c (2) +d where c and d are 


A 
some constants that can be found from the boundary conditions: P, = 0 = c (2) +dandPz,=1= 


1 C3 
app and d = —7——z, and thus, P; = A 


() -@) G).-@) Oo 


Now assume = = 1. We look for the solution of the recurrence relation in the form P; = ci + d. 


Cc () +d. From here, c = — 


Again, from the boundary conditions, P, = 0 = cA +d and Pg = 1=cB +d. Hence, 
= — andd =—— , leading to P; = —* 
mS ee ee 8 L B-A‘ 


(b) By conditioning on the first step, we see right away that the expectation satisfies the recurrence 
relation E; = pE;,, + qE;_, +1 with the boundary conditions E, = Ez = 0. Because of the additive 
constant term, this equation is referred to as a non-homogeneous relation and the general solution is 
i Fi 
sought in the form E; = c (2) +d+ ree if #1, and E; = ci+d-—i?,if ; = 1. The constants c 
and d are found from the boundary conditions. In the former case, 
A B 
they satisfy E, =0 = (2) +d+—and E,=0=c (2) de Whence, 
p q-p Reed p q-p 


GC. —_——= 


and 


resulting in 
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oy 


i 


In the latter case, c and d solve E, = 0 = cA +d — A? and Ez = 0 = cB +d — B”. From here, 
c= A+B andd = —AB. Thus, FE; = (A+ B)i— AB —i2 = (B—i)(i—A). 


(c) We use the formulas derived above with p = 0.47,q = 0.53,A = 10,i = 40, and B = 80. We 


obtain 
10 


0.53 0.53 
Pe (ccs Mao cd 


10 80 
(sz7) - (27) 


40 


and 


0.53\*9 0.53)" 
_ 80-10 (G47) -(ca7) __80- Seer 
40 0.53 — 0.47 Cy or 0.53 — 0.47 " " 


0.47 0.47 


The probability of doubling the fortune in this rigged game is very small (about 0.008), and the 
gambler will play, on average, about 491 games before he walks out of the casino. 


Below we give the code that simulates 10,000 trajectories and computes the proportion of them that 
ended in $80 (as opposed to $10) and averages the number of games in each trajectory. 


#specifying parameters 
p<- 0.47 
ntraj<- 10000 


setting seed number 
set.seed (314159) 


defining walk as matrix 
walk<- data.frame (NULL) 


setting counters 
n80<- 0 

nl10<- 0 

ngames<- 0 


#Simulating trajectories 
fOr (jin Lintray) 4 
walk[1,j3]<- 40 
k<- 2 
repeat { 
walk[k,j]<- ifelse(runif(1)<p, walk[k-1,j]+1, walk[k-1,j3]-1) 
if (walk[k,j]==80) { 
n80<- n80+1 
break 
} 
if (walk[k,j]==10) { 
nl10<- nlO+1 
break 
} 
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k<- k+l 
ngames<- ngames+1l 
} 

} 


print (prop.n80<- n80/ntraj) 
0.0084 
print (avg.ngames<- ngames/ntraj) 


488.4926 


EXERCISE 2.8. The student’s visit to the museum can be modeled as a random walk on a graph 
with the state space S = {Exit, A,B,C, D,E, F}, and the transition probability matrix P 


Exit A B C D &E 
0 oO O 0 

0 1/3 O 1/3 0 

1/2 0 1/2 0 0 

0 1/3 O 1/3 1/3 


COCO OOF 


0 1/2 0 0 
0 1/2 0 0 1/2 
0 60 1 0 


We assume that at the beginning of the walk, the student enters the museum and finds himself in 
Room A. The expected number of transitions between the rooms until he reaches the exit is given by 
the formula 
EG of transitions) 
= (0,1, 0,0,0, 0,0)((1)(P) + (2)(P* — P) + (3)(P? — P*) + (4)(P* — P?) 
+++) (1,0,0,0,0,0,0)~? 


We submit the following R code that approximates this sum. The convergence is achieved with 146 
terms. 


#specifying the transition probability matrix 

tm<- matrix(c(1,0,0,0,0,0,0,1/3,0,1/3,0,1/3,0,0,0,1/2,0,1/2,0, 
1/3;0,1/37 173,070,172, 0, 17270507070; 0,07 1/2;,.0;,-0;,- 1/2, 0,0; 00505 
nrow=7, ncol=7, byrow=TRUE) 


#setting counter 
ntrans<- 0 


#computing expected number of transitions 
p<- matrix(NA, nrow=146, ncol=7) 
pli, ]<- c(0,1, 0, 0, 0, 0, 0) 


for (i in 2:146) { 
pli, ]<- pli-1,]3*3tm 
ntrans<- ntrans+(i-1)*(p[i,1]-p[i-1,1]) 


} 
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print (ntrans) 
12.94225 
Thus, the student will make, on average, 12.94225 transitions between the rooms. Since he spends 30 


minutes in each room, the total average length of visit will be (30)( 12.94225)=388.2675 minutes (or 6 
hours and 28.3 minutes). On an “average” visit, he will be done before the museum closes for the day. 
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CHAPTER 3 


EXERCISE 3.1. We use the independence and stationarity of increments of a Poisson process to 
derive the expression for the joint probability distribution. We write 


P(N(s) =m, N(t) = n) = P(N(t) — N(s) =n —™m,N(s) =m) 
= P(N(t) — N(s) =n—m)P(Ns) =m) = P(N(t —s) =n—m)P(Ns) =m) 


_ Ue ~ s))"" e A(t-s) F (As)™ eAs = (t — ors Are 4t 
(n-—m)! m! (n-—m)!m! 


EXERCISE 3.2. Assume that s < t. We compute the covariance function, using the independence 


and stationarity of the increments. We have 
Cov(N(s), N(t)) = E[N(s)N(t)] — E[N(s)]E[N()] 
= E[(N(t) — N(s) + N(s))(N(s))| — EIN(sS)]JEIN@] 
= E[(N(t) — N(s))N(s)] + E[N(s)]? — E[N(s)]E[N(0)] 
= E[N(t) — N(s)]E[N(s)] + Var[N(s)] + EW) — E[N(s)]E[N(t)] 
= E[N(t — s)]E[N(s)] + Var[N(s)] + [E(N(s))] — E[N(s)]E[N(t)] 
= A(t —s)As + As + (As)? — AsAt = As. 


EXERCISE 3.3. (a) P(N(5) = 16| N(1) = 2,N(2) — N(1) = 3) 


_ P(N(5) — N(2) = 11,N(2) — N(1) = 3,N(Q) = 2) 
7 P(N(2) — N(1) = 3, N(1) = 2) 


_ P(N(3) = 11)P(N(1) = 3)P(N(A) = 2) | ree (16) 
> P(N(1) = 3)P(N(1) = 2) se ali 11! 


(b) Since E(Sj99) = (100) (=) = 20, the 100th claim is expected to be seen on the 20th business 
day, that is, on January 27th. 


Monday 2. |9 16 | 23_-| 30 
Tuesday 3 10 {17 =| 24 [31 
Wednesday | 4 11 18 | 25 
Thursday =§ 5 

Friday 6 

Saturday 7 

Sunday 8 IS 2 || 28 


e~)(3) = 0.0663. 
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EXERCISE 3.4. (a) Phone calls that result in sales occur with rate (0.15) (=) = 1.8 per hour. 


Therefore, in the next two hours, there will be, on average (2)(1.8)=3.6 successful sales. 


(b) The total number of phone calls is a Poisson process with a rate of 60/5=12 per hour. Phone calls 
that result in a sale and those that don’t form independent Poisson processes with rates 1.8 and 10.2 
per hour, respectively. Therefore, 


P(N(1) = 15, Negie(1) = 5) = P(Neaie(1) = 5, Nno sate(1) = 10) 
(18) 2 0.2)72 
! 


= P(Ngqte(1) = 5)PCNno sate(1) = 10) = —>— e7 10! 


5 e102 = 0.00325. 


(c) P(N(4) = 10| N(1) = 3) = P(N(4) — N(1) = 7) = P(N(3) = 7) = (A9G)" 6-A.9)@) = 0.119987. 


EXERCISE 3.5. (a) N,(t) and N,(t) are splitted Poisson processes with the means 


t 


E(N,(t)) = af P(disease is contracted at time s, symptoms show by time t)ds 


0 
t t 


=a r@e-s)ds=(u=t-s}=2] FoW)du, 
and : ° 


t 
E(N2(t)) = af P(disease is contracted at time s, no symptoms show by time t)ds 
0 
t t 
= | (1-F(t—-s))ds ={u=t--s} =a] ¢! — F(u))du. 
0 0 


(b) Suppose by a fixed time t, F (N, (t)) individuals are observed who show symptoms of a 
E(N,(t)) 
i; F(u)du 
into the expression for the expected value of N2(t), we can calculate the estimated number of 
individuals infected but not yet showing symptoms by time t as 


disease. From here, we can estimate the rate of contracting the disease as J = . Plugging this 


B(N,(t)) f,(4- F())du 


E(N2(t)) = je FCadu 


(c) Suppose the incubation period until symptoms show is an exponentially distributed random 
variable with a mean of 2 days. Thus, F(u) = 1— e~“/?,u > 0. Given that E(N,(10)) = 1000, we 
estimate the number of individuals who are infected but haven't shown the symptoms yet as 


10 
1000 f?°e"Zdu _ (1000)(2)(1-e~? ) 


10 


EWG) = f(1- e2) du 10—(2) (1 = e7) 


= 247.8979, 
or about 248 individuals. 
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EXERCISE 3.6. (a) Let N(t) denote the number of high road surface distress areas on a t-mile 
stretch of the road. It is a Poisson process with a rate A = 2.8. So, E(N(10)) = (2.8)(10) = 28. 


(b) The code below simulates 30 distances between distressed surface areas. These distances are 
independent and exponentially distributed with mean = =0.357143 miles. 


#specifying parameters 
lambda<- 2.8 
Nareas<- 30 


defining states 
N<- 0O:Nareas 


setting distance as vector 
dist<- c() 


setting initial value for distance 
dist[1]<- 0 


specifying seed 
set.seed (777754) 


for (i in 2: (Nareas+1) ) 
dist[i]<- dist[i-1] + round((-1/lambda) *log(runif (1)),2) 


#plotting trajectory 
plot(dist, N, type="n", xlab="Number of miles", ylab="Number of distressed areas", 
panel.first = grid()) 


segments (dist [-length(dist)],N[-length(dist)], dist[-1]-0.07, N[-length(dist)], 
lwd=2, col=4) 


points(dist, N, pch=20, col=4) 
points (dist[-1],N[-length(dist)],pch=1, col=4) 


156 20 25 30 


Number of distressed areas 
10 


0 5 10 15 


Number of miles 
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In this simulation, the total length of the road that contains 30 distressed surface areas is the last value 


in the vector dist, that is, 15.99 miles. 
dist [length (dist) ] 


15.99 


(c) Given that N(10) = 30, the distances between distressed surface areas are distributed as order 


statistics of the uniform distribution on the interval (0,10). The R code below simulates the locations 


of those areas. 


specifying parameters 
D<- 10 
Nareas<- 30 


specifying seed 
set.seed (87998) 


defining states 
N<- O0:Nareas 


generating N standard uniforms 
u<- c() 
u[1]<- 0 


for(i in 2: (Nareast1) ) 
u[i]<- runif (1) 


#computing event distances 
dist<- D*sort (u) 


#plotting trajectory 


plot(dist, N, type = "n", xlab="Number of miles", 


areas", panel.first = grid()) 


segments (dist [-length(dist)],N[-length(dist)], dist[-1]-0.07, N[-length(dist)], 


lwd=2, col=4) 


points(dist, N, pch=20, col=4) 


points (dist[-1],N[-length (dist) ],pch=1, col=4) 
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EXERCISE 3.7. For the data set on significant volcanic eruptions between 1920 and 2020, the code 
below calculates interarrival times, plots a histogram, and conducts the goodness-of-fit test. The p- 
value for the test is larger than 0.05, indicating that the Poisson process models the data well. 


volcanoes.data<- read.csv(file="./volcanoesdata.csv", header=TRUE, sep=",") 


#creating date-time variable 
datetime<- as.POSIXct (paste (as.Date (volcanoes.data$SDATE), volcanoes.dataSTIM 


Gl 
~~ 
~~ 


#computing lag 
datetime.lag<- c(0,head(datetime, -1)) 


#computing interarrival times (in hours) 
int<- (as.numeric (datetime) -as.numeric(datetime.lag) )/(3600*24) 
int<- int[-1] # removing first value 


#plotting histogram 
hist (int, main="", xlab="Interarrival Time", col="dark magenta") 
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#binning interarrival times 

binned.int<- as.factor(ifelse(int<25,"1", ifelse(int>=25 & int<50,"2", 
ifelse(int>=50 & int<100,"3", ifelse(int>=100 & int<150,"4",ifelse(int>=150 & 
int<200,"5", ifelse(int>=200 & int<250,"6", "7"))))))) 


#computing observed frequencies 
obs<- table (binned.int) 


#estimating mean for exponential distribution 
mean.est<- mean(int) 


#computing expected frequencies 

exp<- c(1:7) 

exp[1]<- length (int) * (l-exp(-25/mean.est) ) 

exp[2]<- length (int) * (exp (-25/mean.est) -exp (-50/mean.est) ) 
exp[3]<- length (int) * (exp (-50/mean.est) -exp (-100/mean.est) ) 
exp[4]<- length (int) * (exp (-100/mean.est) -exp(-150/mean.est) ) 
exp[5]<- length (int) * (exp (-150/mean.est) -exp (-200/mean.est) ) 
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exp[6]<- length (int) * (exp (-200/mean.est) -exp (-250/mean.est) ) 
exp[7]<- length (int) *exp (-250/mean.est) 


obs 


1 2 3 4 5 6 7 
96 64 94 42 29 13 33 


round (exp,1) 


83.3 64.6 88.9 53.5 32.2 19.3 29.2 


#computing chi-squared statistic 
print (chi.sq<- sum((obs-exp) *2/exp) ) 


7.581284 


#computing p-value 
print (p.value<- l-pchisq(chi.sq, df=5) ) 


0.1808718 


EXERCISE 3.8. (a) Team A scores as a Poisson process with a rate A, = (0.25 + 0.40 + 
0.20)(0.5) = (0.85)(0.5) = 0.425 per minute. Team B scores as a Poisson process with a rate Ap = 
(0.25 + 0.50 + 0.15)(0.4) = (0.9)(0.4) = 0.36 per minute. Hence, 


‘i 
AatdAg _(0.425+0.36 


= 1.273885 minutes. 


E(time until a team scores) = 


(b) E(time until team A scores) = + = —+_ = 2.352941 minutes. 
Aa 0.425 


E(time until team B scores) = 7 = = 2.777778 minutes. 
B . 


aa 0.425 
AatAp _—(0.425+0.36 


(c) P(team A scores before team B) = = 0.541401, 


P(team B scores before team A) = 1 — 0.541401 = 0.458599. 


(d) For team A, 1-pointers occur as an independent Poisson process with rate (0.25)(0.5) = 0.125 per 
minute; 2-pointers occur as an independent Poisson process with rate (0.4)(0.5) = 0.2 per minute; 
and 3-pointers occur as a Poisson process with rate (0.20)(0.5) = 0.1 per minute. For team B, the 
respective rates are (0.25)(0.4) = 0.1 per minute, (0.5)(0.4) = 0.2 per minute, and (0.15) (0.4) = 
0.06 per minute. Therefore, 


P(teams score same #of 1-pointers, 2-pointers, and 3-pointers) 
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= P(same #of 1-pointers)P(same #of 2-pointers)P (same #of 3-pointers) 


peas einatng ((0.125)(0.1)(48)7)"| | _cyonoaycae) Wo ((0-2)(0.2)(48)?)" 
=| 5+0.1)( ”y aa? | |e ( Ne yy oe ae 


x e- 1+0.06)(48) 2 eee B) = 0.0012. 


#computing probability of same number of 1-pointers, 2-pointers, and 3-pointers 
suml<- 0 
for(n in 0:16) 
suml<- sum1+(0.125*0.1*48%2) “n/ (factorial (n))%*2 
print (pl<- suml*exp(-(0.125+0.1) *48) ) 
0.1152988 
sum2<- 0 


for(n in 0:23) 
sum2<- sum2+(0.2*0.2*48%2) “n/ (factorial (n))%*2 


print (p2<- sum2*exp(-(0.2+0.2) *48) ) 
0.09165684 
sum3<- 0 
for(n in 0:12) 
sum3<- sum3+(0.1*0.06*48%2) *n/ (factorial (n))%*2 
print (p3<- sum3*exp(-(0.1+0.06) *48) ) 


0.1167362 


pl*p2*p3 


0.001233658 


EXERCISE 3.9. (a) The spider will need tT = 30 minutes = 0.5 hours to reach the top. The rate of 


rain is A = 2 per hour. Denote by T the total time it takes the spider to reach the top. The expected 
value of T is E(T) = = (e5)@) — 1) = 0.859141 hours or 51.5 minutes. 


(b) Let N denote the number of times the spider will be washed down before it reaches the top. Then, 


E(N) = e@)@ — 1 = 1.718282. 
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CHAPTER 4 


EXERCISE 4.1. (a) The number of broken calculators can be modeled according to a 
nonhomogeneous Poisson process {N(t), t = 0} with the intensity rate function 


3, if 0<t <3, 
Ole aes if 3<t <10. 


The integrated rate function is 


t 
‘ | sau =3, if O<t <3, 
a = | A(ujdu = <°° . 
. 94+ | (2u-3)du= 1-36-49, if 3<t <10. 
3 


The probability mass function is 


P(N(t) — N(s) =n) = BOAR" eAO-AG)) = CP SEO) pr (e-s?-300-8)), 


50 
(b) P(N(8) — N(4) = 50) = aa = 0.004983. 


(c) E(N(10) — N(2)) = A(10) — A(2) = 102 — (3)(10) + 9 — (3)(2) = 73. 


EXERCISE 4.2. (a) Below is the R code and plot of the intensity function A(t) = —0.000025 t? + 
0.002 t? + 0.12t against t on the interval [0, 120]. 


lambda<- function(t) -0.000025*t%*3+0.002*t%2+0.12*t 
t<- seq(0, 120, by = 0.01) 


plot(t, lambda (t) ) 


0 20 40 60 80 100 120 
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The intensity function looks like a skewed upside-down parabola that is equal to 0 at t = 0 andt = 
120. It achieves the maximum at time t that solves the equation A'(t) = 0. Therefore, t solves the 
quadratic equation (—0.000025)(3)t? + (0.002)(2)t + 0.12 = 0, which simplifies to —0.0075t? + 


0.4t +12 = 0. We need the solution that lies between 0 and 120, therefore, t = a = 
74.74068 days. Thus the peak intensity rate occurs 74.74 days into the fire season, and the maximum 
number of fires per day is = —0.000025 (74.74068)? + 0.002 (74.74068)? + 0.12(74.74068) = 


9.703286, or roughly 9.7 fires per day. 


(b) We write A(t) = f, A(u)du = f° (—0.000025 u3 + 0.002 u? + 0.12u)du = — 0.00000625¢* + 
0.0006667t? + 0.06t7,0 <t < 120. The R code and the graph are given below. 


Lambda<- function(t) -0.00000625*t%*4+0.0006667*t%*3+0.06*t%*2 
t<- seq(0, 120, by = 0.01) 


plot(t, Lambda (t) ) 


Lambdai(t) 


0 20 40 60 80 100 120 


To find the average number of wildfires per season we compute A(120) = —(0.00000625)(120)* + 
(0.0006667)(120)? + (0.06)(120)? = 720.0576, so, on average, about 720 fires occur in this area 
every season. 


(a) The middle 50% of the fire season falls between day 30 and day 90. The mean number of wildfires 
in this period is A(90) — A(30) = —(0.00000625)(90)* + (0.0006667)(90)? + (0.06)(90)? — 
(—(0.00000625)(30)* + (0.0006667)(30)? + (0.06)(30)7) = 561.9618 — 66.9384 = 495.0234. 


EXERCISE 4.3. (a) The integrated intensity function is 


AC) = f, Aw)du = Joa du = 2AVt,t > 0. 
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We know that A(1) = 30. Thus, A = 15. 
(b) Given that the nth injury occurred at the time s,, the conditional cumulative distribution function 


of the time until the next injury T,,1,n = 1, is Fr, ,.)5,(tlSn) = 1—- exp{—30(,/t + Sy - 
VSn)} ,t = 0. The cdf of T, is Fr, (t)=1- exp{—30v¢}, t > 0. To simulate observations from 
this distribution, we first generate standard uniform random variables U,, n = 1, and then 
sequentially solve Fr, js, (tlSn) = Un+1, i-€., we solve for t the identities 1 — 


exp{—30(,/t mee ieee VSn)} = U,. Replacing 1 — U,, by U, since both have the same standard 
uniform distribution, we obtain 


2 
) 


S,=(-ZinW,)), and Sys = (Sp - A in(U,)) on > 1, 


The code and the simulated trajectory follow. 


specifying parameters 
ninjrs<- 100 


defining states 
N<- O:ninjrs 


defining times as vectors 
time<- c() 


specifying seed 
set.seed (933541) 


computing event times 

time[1]<- (-log(runif(1))/30)%*2 

for(i in 2: (ninjrstl)) 

time[i]<- (sqrt (time[i-1])-log(runif(1))/30)%*2 


#plotting simulated trajectory 
plot(time, N, type="n", xlab="Years", ylab="Number of injuries", 
panel.first=grid()) 


segments (time[-length(time)], N[-length(time)], time[-1]-0.07, 
N[-length(time)], col=4) 


points (time, N, pch=20, col=4) 
points (time[-1], N[-length(time)], pch=1, col=4) 
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The time range of the trajectory is [0, 9.870481]. 


(c) Weare given that S199 = 12.25 years. The times of injuries S,, S,...,S9q are then represented by 
A(s) _  30VS) 
A(S100)  30/S100 


order statistics from the distribution with the cumulative distribution function 


4/ S/S199. The code that simulates a trajectory and the graph follow. 


t<= 12.25 
Lambda<- 30*sqrt(t) 


#generating N(t) 
set.seed (1133664) 
ninjrs<- rpois (1, Lambda) 


defining states 
N<- O:ninjrs 


generating standard uniforms 
u<- c() 

u[1]<- 0 

for(i in 2: (ninjrs+1) ) 

u[i]<- runif (1) 


computing event times 
time<- round(t*sort (u) *2,4) 


plotting simulated trajectory 
plot(time, N, type="n", xlab="Years", ylab="Number of injuries", 
panel.first=grid()) 
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segments (time[-length(time)], N[-length(time)], 


N[-length(time)], col=4) 


points (time, N, pch=20, col=4) 
points (time[-1], N[-length(time) ], 


40 60 80 =6100 


Number of injuries 


20 


simulation method where we uniformly dominate the intensity rate function by 20. 


specifying parameters 


lambda.star<- function(s) 20 
Lambda.star<- function(s) 20%*s 


specifying seed 
set.seed (2866514) 


generating N(t) 
njumps<- rpois(1, Lambda.star(10) ) 


generating N(t) standard uniforms 
u<- c() 
u[1l]<- 0 


for(i in 2: (njumps+1) ) 
uf[i]<- runif (1) 


computing event times 
time.star<- 10*sort(u) 


thinning event times 
accepted<- c() 

time<- c() 
accepted[1]<- 1 
time[1]<- 0 


for (i in 2:(njumps+l)) { 


pch=1, 


col=4) 


Years 
EXERCISE 4.4. Below are the code and plot of the simulated trajectory. We are using the thinning 


lambda<- function(s) 10+10*cos (2*pi*s) 


time[-1]-0.07, 


if (runif(1)<= lambda(time.star[i])/lambda.star(time.star[i])) 
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accepted[i]=l1 else accepted[i]=0 
} 


time<- time.star[-which (accepted==0) ] 
N<- 0: (length (time) -1) 


#plotting trajectory 
plot(time, N, type="n", xlab="Time", ylab="State", panel.first = grid()) 


segments (time[-length(time)],N[-length(time)], time[-1]-0.07, 
N[-length(time)], col=4) 


points(time, N, ylim=c(0,120), pch=20, col=4) 
points (time[-1],N[-length (time) ],pch=1, col=4) 


100 
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Time 


EXERCISE 4.5. In the process of radioactive decay, photons are emitted according to a 
nonhomogeneous Poisson process with the intensity rate A(t) = 100e~°*, t => 0. The integrated 


intensity rate function is A(t) = f, A(s)ds = 20001 —2e""), ¢ = 0, 


METHOD 1 (EXPONENTIAL INTERARRIVALS). To simulate event times, we solve the recurrence 
equations 1 — e~4G0 = U, and 1 — e~(AGn+2)-AGn)) = YU, ,,,n > 1. Equivalently, we can replace 
1 — U, by U; and solve e460) = U, and e~(AGn+1)-AGn)) = Uns41,n = 1. In this example, the 
equations are: e~200@-e° 1) = Y, and e~2000e Pn-e P5541) _ Yn > 1. Solving, we get 


S, = —2In (1 # (—) In(U;)) and S,4, = —2In Ga (—) In(Un+1)) nei. 


The code below simulates a trajectory with 20 events. 


specifying parameters 
njumps<- 20 


defining states 
N<- O:njumps 


defining times as vectors 
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time<- c() 


specifying seed 
set.seed (40556002) 


generating standard uniforms 
u<- c() 

for(i in l:njumps) 

u[i]<- runif (1) 


computing event times 
time[1l]<- 0 
time [2]<- -2*log(1+(1/200) *log(u[1]) ) 


for(i in 3:(njumps+l)) { 
time[i]<- -2*log(exp(-0.5*time[i-1])+(1/200) *log(u[i-1])) 
} 


plotting trajectory 

plot(time, N, type="n", xlab="Time", ylab="State", panel.first = grid()) 
segments (time[-length(time)],N[-length(time)], time[-1]-0.001, N[-length(time) ], 
lwd=2, col=4) 


points(time, N, ylim=c(0,120), pch=20, col=4) 
points (time[-1],N[-length(time)], pch=1, col=4) 
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METHOD 2 (UNIFORM ORDER STATISTICS). In this method, an event time S is found as the 
solution of the equation NCES = U where by U we denote a standard uniform random variable from 


200(1-e7 955) 
200(1—-e—(-5)(0.25)) 
—2In(1 — (1 — e~@5)@-5))U), The code and graphical output are presented below. 


an ordered sample. The equation takes the form: = U. The solution is § = 


#specifying parameters 
t<- 0.25 
Lambda<- 200* (l-exp(-0.5*t) ) 
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specifying seed 
set.seed (492231) 


generating N(t) 
njumps<- rpois (1, Lambda) 


defining states 
N<- O:njumps 


generating N(t) standard uniforms 
u<- c() 
u[1]<- 0 


for(i in 2: (njumps+1) ) 
u[i]<- runif (1) 


#computing event times 
time<- -2*log(1-(l-exp(-0.5*t)) *sort (vu) ) 


#plotting trajectory 

plot(time, N, type="n", xlab="Time", ylab="State", panel.first = grid()) 
segments (time[-length(time)],N[-length(time)], time[-1]-0.001, N[-length(time) ], 
lwd=2, col=4) 


points(time, N, ylim=c(0,120), pch=20, col=4) 
points (time[-1],N[-length(time)], pch=1, col=4) 
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METHOD 3 (THINNING). We bound the intensity rate function A(t) = 100e~°*", t > 0, uniformly by 
100. The code below simulates events times of a homogeneous Poisson process and then applies the 
algorithm of the thinning method to select only those event times that belong to the nonhomogeneous 
Poisson process. The plot follows. 


#specifying parameters 

lambda<- function(s) 100*exp(-0.5%*s) 
lambda.star<- function(s) 100 
Lambda.star<- function(s) 100*s 


#specifying seed 
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set.seed (2866514) 


#generating N(t) 
njumps<- rpois(1, Lambda.star(0.25) ) 


#generating N(t) standard uniforms 
u<- c() 
u[1l]<- 0 


for(i in 2: (njumps+1l) ) 
uf[i]<- runif (1) 


computing event times 
time.star<- 0.25*sort (u) 


thinning event times 
accepted<- c() 

time<- c() 
accepted[1]<- 1 
time[1]<- 0 


for (1 in 2:(njumps+l)) { 
if (runif(1)<= lambda(time.star[i])/lambda.star(time.star[i])) 
accepted[i]=l1 else accepted[i]=0 
} 
time<- time.star[-which (accepted==0) ] 


N<- 0: (length (time) -1) 


plotting trajectory 

plot(time, N, type="n", xlab="Time", ylab="State", panel.first = grid()) 
segments (time[-length(time)],N[-length(time)], time[-1]-0.001, N[-length(time) ], 
lwd=2, col=4) 


points (time, N, ylim=c(0,120), pch=20, col=4) 
points (time[-1],N[-length(time)], pch=1, col=4) 
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EXERCISE 4.6. (a) We run the following code to plot the counts of lightning deaths against year. 


data<- read.csv(file="./lightningdata.csv", header=TRUE, sep=",") 


plot (dataSYEAR, dataSFREQ) 
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dataSYEAR 


We can see from the plot that the intensity rate decreases over time roughly exponentially. A possible 
explanation for it is that over the years more awareness has been created among citizens through 
educational efforts, so fewer people are exposed to the hazard. 


(b) We can see that lightning strikes are essentially a seasonal phenomenon. The majority of them 
happen between May and September. It means that some interarrival times have very large values not 
inherent to an exponential distribution. Moreover, some incidents resulted in multiple fatalities which 
would be an event of probability zero under the Poisson law. 


EXERCISE 4.7. (a) The code and output below estimate the parameters of the model using the 
regression approach. 


port.data<- read.csv(file="./Exercise4.4Data.csv", header=TRUE, sep=",") 


x<- log (port.dataSdays) 
y<- log(port.dataSarrivals) 


glm(y~x) 


plot(x,y, xlab="lIn(days)", ylab="l1n(arrivals)") 


Coefficients: 
(Intercept) x 
-1.294 1.145 


lines(x, -1.294+1.145*x) 
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In(arrivals) 
1.0 16 2.0 25 3.0 


0.5 


0.0 
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In(days) 


The estimates of the model parameters are @ = e~1?9* = 0.274172, and B = 1.145. 


(b) The code that follows estimates the parameters using the maximum likelihood approach. 


port.data<- read.csv(file="./Exercise4.4Data.csv", header=TRUE, sep=",") 


x<- log(port.dataSdays) 

y<- log(port.dataSarrivals) 

N<- 27 

print (beta.hat<- N/ (N*x[N]-sum(x) )) 
1.162782 

print (alpha.hat<- N/exp(x[N]*beta.hat) ) 


0.2351766 


The MLEs are @ = 0.2351766, and B = 1.162782. 


(c) To predict when the next 10,000 TEUs arrive at the port, we submit the following lines of code. 


58 


port.data<- read.csv(file="./Exercise4.4Data.csv", header=TRUE, sep=",") 


x<- log (port.dataSdays) 
y<- log(port.dataSarrivals) 
N<- 27 


alpha.hat<- c(0.274172, 0.2351766) 
beta.hat<- c(1.145, 1.162782) 
S.hat<- c() 


library (pracma) 


for (i an Ps2) 

print (S.hat[i]<- alpha-hat[i]%*(- 

1/beta.hat[i]) *exp(alpha.hat[i]*exp(x[N])*beta.hat[i])* 
gammainc(alpha.hat[i]*exp(x[N])*beta.-hat[i], 1/beta.hat[i]+1) [2]) 


60.85581 
60.97308 


According to the data, the 27th arrival was on day 59.1. The 28th arrival is predicted to be on day 
60.85581 (by the linear regression), or 60.97308 (by the maximum likelihood). 
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CHAPTER 5 


EXERCISE 5.1. (a) Let X(t) = pee Y; be the total amount paid in prizes up to time t hours. We 
know that it is a compound Poisson process with N(t)~Poisson(1.5t), and Y; independent of each 
other and N(t). The first two moments of Y, are E(Y,) = ($5000)(0.15) + (2000)(0.35) + 

($500) (0.2) + ($100)(0.3) = $1,580, and E(Y2) = ($5000)2(0.15) + (2000)2(0.35) + 


($500)(0.2) + ($100)2(0.3) = $25,203,000. 


Therefore, the mean of X(200) is E(X(200)) = (1.5)(200)($1,580) = $474,000. The variance is 
Var(X(200)) = (1.5)(200) ($75,203,000) = $71,560,900,000, and the standard deviation is 


[Var(x(200)) = ,/$21,560,900,000 = $39,508.23. 


The budget for 100 games should be E(X(200)) + [var(x(200)) = $474,000 + $39,508.23 = 
$513,508.23. 


(b) Below are the codes, all relevant output, and the graph for the simulated 100 games. 


#specifying parameters 

lambda<- 1.5 

total.hours<- 200 

amount<- ¢c(5000, 2000, 500, 100) 
ps] ©0215) (023 55- On2) O38) 


specifying seed 
set.seed (704661) 


generating number of prizes 
nprizes<- rpois(1,lambda*total.hours) 


defining vectors 
payoff<- c() 
time<- c() 

u<- c() 


setting initial values 
payoff[1]<- 0 
u[l]<- 0 


generating standard uniforms 
for(i in 2:(nprizes+l)) { 
u[i]<- runif (1) 
payoff[i]<- payoff[i-1] + amount[sample(1:4, 1, prob=p) ] 
} 


#computing event times 
hour<- total.hours*sort (u) 


#simulating trajectory 


plot(hour, payoff, type="1", lty=1, lwd=2, col="green", xlab="Hours", ylab="Amount 
of payoff ($)", panel.first = grid()) 
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payoff[length (payoff) ] 
557100 


There were a total of 308 prizes given out during the 200 hours of the 100 games. The total payoff was 
$557,100.00. Since the budget that the producer had for the 100 games was $513,508.23, the producer 
ran out of money before the 100th game. 


payoff 
[289] 511200 513200 513300 515300 515400 
The producer ran out of budget after the 291st prize was given out. 


hour [291] 
190.8272 


The 29 1st prize was given out at hour 190.8272 of the show, that is, during the first half of the 96th 
show. 


EXERCISE 5.2. (a) Let ¥;~Unif ($30, $300) be the ith claim amount. The two first moments of Y; 


3 3 
seb) = $304+8300 = $165, and E(¥,”) = 4 $300 a ($300)*—($30)" _ $233 300. 


$300-$30 “$30 3($270) 
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Let X(t) = eg Y; be the aggregate claim process. Its mean for t = 30 is 


E(X(30)) =(60)(30)($165)=$297,000, and the standard deviation is /Var(X(30)) = 


(60) (30) ($233,300) = $7,742.093. 


(b) By the Central Limit Theorem, Z = See ee 


has approximately a N(0,1) 


Var(x(30)) —«7742.093 
distribution, and therefore, P(X(30) > 300000) = P (z > a) = P(Z > 0.387492) 


0.349196. 


EXERCISE 5.3. (a) Let X(t) = yo Y; denote the aggregate number of light photons that are 


generated up to t seconds. We are given that N(t)~Poisson(At) and ¥,~Poisson(A). The mean and 


standard deviation of X(t) are E(X(t)) = (A)(t) (A) and Var(X (t)) =i (A)(E)(EY?) = 


(aye (a + 22). 


(b) The code below simulates 100 values of the aggregate number of light photons generated within 10 


seconds. 


#specifying parameters 
total.time<- 10 
lambda<- 50 
lambda.tilde<- 5 


total.photons<- c() 
for (j in 1:100) { 


nphotons<- c() 
nphotons[1]<- 0 


#generating N(t) 
set.seed(150*3) 
N<- rpois(1,lambda*total.time) 


#simulating trajectory 
for (i in 2:N) 
nphotons[i]<- nphotons[i-1]+rpois(1,lambda.tilde) 


total.photons[j]<- nphotons[N] 


total.photons 
[1] 2333 2737 2532 2392 2612 2484 2474 2440 
[9] 2638 2549 2549 2361 2459 2603 2633 2537 
[17] 2379 2314 2496 2630 2410 2233 2350 2610 
[25] 2506 2412 2444 2444 2572 2430 2477 2472 
[33] 2538 2470 2648 2641 2333 2425 2475 2549 
[41] 2382 2599 2451 2386 2477 2262 2673 2627 
[49] 2493 2490 2789 2449 2572 2537 2566 2427 
[57] 2591 2469 2414 2508 2610 2587 2523 2384 
[65] 2695 2578 2633 2523 2355 2597 2577 2558 
L73] 2550 2396 2406 2496 2576 2627 2434 2552 
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[81] 2425 2393 2504 2664 2485 2408 2625 2500 
[89] 2595 2609 2521 2524 2420 2569 2659 2741 
[97] 2360 2617 2546 2505 


(c) Below we construct a histogram for these 100 values. 


hist (total.photons, col="maroon") 


Histogram of total.photons 
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The histogram does resemble a bell shape. It should be the case since A is large, and so, the Central 
Limit Theorem should be applicable. 


EXERCISE 5.4. The average present value of the total claim amount is computed by conditioning 
on the value of N(t). We proceed as follows: E[P(t)] = EE[P(t)|N(t)] = EE ey met | N(t)| 
=E pS E(X,E(e7% |. The claim arrival times $y, ..., Sy¢¢) are order statistics from a uniform 
distribution on [0, t] and we can write S; = tU(j) where Ui is the ith order statistic from the standard 
uniform distribution. We continue E[P(t)] = E bats E(X,)E (e*4@)] = E(X,)E pala E (ee )| 
= E(X,)E(N(t))E(e78) = E(X,)A f. e-o™du = E(X,) (5) (1 -—e-*), 


EXERCISE 5.5. (a) Random variables Y,'s are iid ~ Poisson (f). Therefore, 
n_.Y; ~ Poisson (Bn). We write P(X(t) = x) = POs Y ¥; = eee = V7) P(r, 
ci x At _ _ n*(at)” _ 
x |N(t) = n)P(N(t) =n) = Dep eon A et =F eit ye ee in 


(b) E(X(t)) = AtE(%,) = AtB, Var(X(t)) = AtEY? = Aat(B + B?). 
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n -B = 7 
(c) P(X(t) = _ 0)= _ e At ye oo (At) e —Bn _ e4t yee aoe eAttAte B = e7At(1-e By 


n! 


The ratio between the variance and mean is ae = = 1+, thus, f can be estimated as 
B= Sean — 1. Also, In P(X(t) = 0) = In P(0) = —At(1 — e~*). Hence, we can estimate A by 
i InP (0) . 

t(1-e7F) 


EXERCISE 5.6. (a) The total dollar amount can be modeled by a compound Poisson process 
X(t) = yas ) Y; where N(t) is the process governing the number of cars that come to the gas station 


up to time t, and Y; is the dollar amount that the ith car driver pays. We are given that 
N(t)~Poisson (At), and ¥,~Gamma(a, B) with mean E(Y;) = @B, and variance Var(Y,) = aB?. 


(b) Denote by T;~Exp(mean = -) the interarrival times between car arrivals. The method of 


moments estimator of A is 4 = 1/T, the reciprocal of the sample mean of the interarrival times. The 


v2 es Vv nm 2 _ v2 
method of moments estimators of a and 6 are @ = nr and B = = te They solve 
the system of two equations: Y = £(Y,) = @B and- an = E(Y?) =Var(%,) + 
(E(%,)) =a? + (a6) = BY +¥?. 


(a) The code below produces numeric values of the estimators and plots histograms with fitted 
curves. 


gas.data<- read.csv(file="./Exercise5.6Data.csv", header=TRUE, sep=",") 


computing lag 
gas.dataSArrivalTime.lag<- c(0,head(gas.dataSArrivalTime, -1)) 
gas.data<-gas.data[-1,] #removing first row 


computing interarrival times 
interarrival.time<- gas.dataSArrivalTime-gas.dataSArrivalTime.lag 


estimating lambda of Poisson arrival 
print (lambda.hat<- 1/mean(interarrival.time) ) 


0.6029832 
print (1/lambda.hat) 


1.658421 


There are, on average, 0.6029832 car arrivals every minute. The average wait time between two 
arrivals is 1.658421 minutes. 


#overlaying histogram and fitted exponential density curve 
hist (interarrival.time, freq=FALSE, col="purple") 

x<- seq(0, 8, by=0.01) 

y<- dexp (x, lambda.hat) 


lines(x, y, lty=1, col="light green", lwd=3) 
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Histogram of interarrival.time 
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#estimating parameters of gamma distribution 

amount<- gas.dataSAmountSpent 

print (alpha.hat<- length (amount) *mean (amount) *2/ (sum(amount%2) - 
length (amount) *mean (amount) “2) ) 


7.48998 


print (beta.hat<- mean(amount) /alpha.hat) 


4.29609 

#overlaying histogram and fitted density 
hist (amount, freq=FALSE, col = "pale green") 
x<- 0:80 


y<- dgamma(x, alpha.hat, 1/beta.hat) 
lines(x, y, lty=1, col="dodger blue", lwd=3) 


65 


Histogram of amount 
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(d) The estimated mean of the total dollar amount at one hour is F (x (60)) — (A) (t) (@)(B ) = 
(0.6029832)(60)(7.48998) (4.29609) = $1,164.154. The estimated standard deviation is 


[Par (x (60) = |(A)(t)(@)(B) = J((.6029832) (60) (7.48998) (4.29609)? = $70.71995. 
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CHAPTER 6 


EXERCISE 6.1. (a) Cov(N(s),N(t) — N(s)) = EIN(s)N(t — s)] — E[N(s)] E[N(t — s)] = 
EE[N(s)N(t — s) | A] — EE[N(s) | A] - EE[N(t — s) | A] = E[(As)(A)(t — s)] — E[As]E[ACt — 
s)] = s(t — s)E(A’) — s(t — s)(E(A))? = s(t — s)Var(A). 


(b) Cov(N(s), N(t)) = E[N(s)N(t)] — E[N(s)] E[N(t)] = EE[N(s)(N(@) — N(s) + N(s)) | A] 
E[N(s)] E[N(t)] = EE[N(s)N(t — s) | A] + EE[(N(s)) | A] — EE[N(s) | A] - EE[N(t) | A] = 
E[(As)(A)(t — s)] + E[As + (As)*] — E(As)E(At) = s(t — s)E(A*) + sE(A) + s?E(A?) — 
st(E(A))” = stE(A*) — st(E(A)) + sE(A) = stVar(A) + sE(A). 


P(N(t)=n,Asa 
EXERCISE 6.2. (a) Fayy(Aln) = P(A < A|N(t) = 7) = eee 


A 


n 
= fo P(N (t) =n | A= U)fa(u)au = i oe Mt fA(u)du = (ume fy (u)du 


iS P(N(t) = n| A= u)fa@au [own out fy (u)du fume“ fy (udu 


=, = Ane-At FAA) 
(b) fatney Alm) = Facey Al) = Fernie, aan’ 
fy ef da 


E[A|N(t) =n] =f°a A\n)da = 
(c) E[AIN(t) =n] = fy Afgan Al) fpr ate “faa 


EXERCISE 6.3. (a) Denote by {N(t), t = 0} the process of visitor arrival. We know that 
N(t)~Poisson(At) where P(A = 4) = 0.46, P(A = 2) = 0.24, and P(A = 3) = 0.30. 

The mean and variance of N(t) are E(N(t)) = tE(A) = t((4) (0.46) + (2)(0.24) + (3) (0.30)) = 
3.22t, and Var(N(t)) = t?Var(A) + tE(A) = t?((4)?(0.46) + (2)?(0.24) + (3)?(0.30) — 
(3.22)2) + 3.22t = 0.6516t? + 3.22t. 


(b) The code below simulates 5 trajectories of the process with 200 visitors each. 


#specifying parameters 
p<- c(0.46, 0.24, 0.30) 
lambda<- c(4, 2, 3) 
nvisitors<- 200 

time<- data.frame() 

N<- data.frame() 


specifying seed 
set.seed (109088) 


creating loop to simulate trajectories 
for(j in 1:5) { 


selecting rate 
Lambda<- lambda[sample(1:3, 1, prob=p) ] 
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#setting initial values 
time[1,j]<- 0 
N[1,j3]<- 0 


#simulating trajectory 
i<- 2 


repeat { 
time [i,j]<- time[i-1,j]+round((-1/Lambda) *log(1l-runif (1)),3)-0.001 
N[i,j]<- N{i-1,3] 


if (i==2*nvisitors+2) break 
else { 
time [itl,j]<- time[i,j]+0.001 
N[(it1,j3]<- N[i,j]+1 
i<- i+2 
} 
} 


} 

#plotting trajectories 

matplot(time, N, type="1", lty=1, lwd=2, col=c("red", "blue", "green", 
"purple", "“orange"), xlab="Minutes", ylab="Number of visitors", 
panel.first=grid()) 
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(c) The code below simulates 5 trajectories of the process that depict arrivals within one 
hour. 


specifying parameters 
t<- 60 

p<- c(0.46, 0.24, 0. 
lambda<- c(4, 2, 3) 
time<- data.frame() 
N<- data. frame () 


30) 


specifying seed 
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set.seed (5055562) 


#Hcreating loop to simulate trajectories 
for(j] in 1:5) { 


selecting rate 
Lambda<- lambda[sample(1:3, 1, prob=p) J 


setting initial values 
time[1,j]<- 0 
N[1,3]<- 0 


generating N(t) 
N.total<- rpois(1,Lambda*t) 


generating N(t) standard uniforms 
u<x- 1:N.total 

for(i in 1:N.total) 

u[i]<- runif(l1) 


sorting standard uniforms 
u.sorted<- sort (u) 


computing N(t) event times 
s<- t*u.sorted 


generating jumps 
for (1 in seq(2, 2*N.total, 2)) { 
time[i,j]<- s[i/2]-0.001 
time [itl,j3]<- s[i/2] 
N[i,j]<- N(i-1,3] 
N[it1,j3]<- N[i-1,j3]+1 
} 
} 


#plotting simulated trajectories 

matplot(time, N, type="1", lty=1, lwd=2, col=c("red", "blue", "green", 
"purple", "orange"), xlab="Minutes", ylab="Number of visitors", 
panel.first=grid()) 
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EXERCISE 6.4. (a) Let N(t) denote the number of defaults by time t. It is given that 
N(t)~Poisson(At) where A~Uniform(0,2). The average number of defaults within 5 years is 


E(N(S)) = tE(A) = (5)(1) =5. 


(b) The variance of the number of defaults within 5 years is Var(N (5)) = t*Var(A) + tE(A) = 
(5)? (= -) + (5)(1) = 13.3333. 


(c) As shown in Exercise 1(a), Cov(N(s), N(t) — N(s)) = s(t — s)Var(A). Fors = 3, andt =5, 
Cov(N(3), N(5) — N(3)) = (3)(5 — 3)(1/3) = 2. 


(d) As shown in Exercise 1(b), Cov(N(s), N(t)) = stVar(A) + sE(A). Fors = 3, andt =5, 
Cov(N(3), N(5)) = (3)(5)(1/3) + 3) (1) = 8. 
Jo? we“ Fa (u)du 


(e) Using the result proven in Exercise 6.2, we get P(A < 0.5| N(5) = 2) = owe FUf (udu = 


05 
Jy ureS4du = . . Bee 
bees Now, f u2e>“du = —<ue et —=ue 5u _ _"_e—5" Therefore, 
Jo uzeSUdu 125 
2,—-5u — Lyte SU 2 ye-Su__2_p-5uj0.s e725 2) 
POA < 0.5 9(S) = 2) = Mp ee = a age age = = aap 
2e—5u ~~ _ 1 2e-su—2ye-su—_2_e—5u = 2 ~ 
se te gee pen. 125° lo° 125 


0.456187. 


EXERCISE 6.5. (a) Let N(t) denote the amount of SWE accumulated within time t. It is given that 
N(t)~Poisson (At) where A~Poisson(A = 24.3). The average and standard deviation of SWE for 


one year are E(N(1)) = tE(A) = At = (24.3)(1) = 24.3 inches, and [Var(N(1)) = 
Vt?Var(A) + tE(A) = vt27A + ta = J (12 + 1)(24.3) = V48.6 = 6.97 inches. For five years, 


E(N(5)) = tE(A) = At = (24.3)(5) = 121.5 inches and_[Var(N(5)) = J (5? + 5)(24.3) = 
V¥729 = 27 inches. 


(b) The code below simulates 5 trajectories that reach 140 inches of SWE each. 


specifying parameters 
lambda<- 24.3 
SWE.inches<- 140 
time<- data.frame() 
N<- data.frame() 


specifying seed 
set.seed (9000004) 


creating loop to simulate trajectories 
FOE An, L5) { 


selecting rate 
Lambda<- rpois (1, lambda) 
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#setting initial values 
time[1,j]<- 0 
N[1,j]<- 0 


#simulating trajectory 
1<- 2 


repeat { 
time [i,j]<- time[i-1,j]+round((-1/Lambda) *log(1l-runif (1)),3)-0.001 
N[i,j]<- N{i-1,3j] 


if (1==2*SWE.inches+2) break 
else { 
time [itl,j]<- time[i,j]+0.001 
N[it1,j3]<- N[i,j]+1 
i<- i+2 
} 


} 


} 

#plotting trajectories 

matplot(time, N, type="1", lty=1, lwd=2, col=c("red", "blue", "green", 
"purple", “orange"), xlab="Years", ylab="Amount of SWE, in inches", 
panel.first=grid()) 
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(c) The code below simulates 5 trajectories spanning over 7 years. 


specifying parameters 
Eat 

lambda<- 24.3 

time<- data.frame() 
N<- data. frame () 


specifying seed 
set.seed (1001117) 
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#creating loop to simulate trajectories 
for(j in 1:5) { 


#selecting rate 
Lambda<- rpois (1, lambda) 


setting initial values 
time[1,j]<- 0 
N[1,j3]<- 0 


generating N(t) 
N.total<- rpois(1,Lambda*t) 


generating N(t) standard uniforms 
u<x- 1:N.total 

for(i in 1:N.total) 

u[i]<- runif (1) 


sorting standard uniforms 
u.sorted<- sort (u) 


computing N(t) event times 
s<- t*u.sorted 


generating jumps 
for (1 in seq(2, 2*N.total, 2)) { 
time[i,j]<- s[i/2]-0.001 
time [it1l,j3]<- s[i/2] 
N[i,j]<- N(i-1,3] 
N[it1,j3]<- N[i-1,j3]+1 
} 
} 


#plotting simulated trajectories 

matplot(time, N, type="1", lty=1, lwd=2, col=c("red", "blue", "green", 
"purple", “orange"), xlab="Years", ylab="Amount of SWE, in inches", 
panel.first=grid()) 
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EXERCISE 6.6. (a) Let N(#) be the number of defects in an f-yard roll of fabric. We know that 


N(#@)~Poisson (Af) where A~Gamma(a, B) with E(A) = a/B = 0.07 and /Var(A) = /a/B2 = 
0.01. The mean and standard deviation of N(40) are E(N(40)) = £E(A) = (40)(0.07) = 2.8 and 


JVar(N(40)) = ./€2Var(A) + €E(A) = /(40)2(0.01)2 + (40) (0.07) = 1.72. 


(b) As derived in Application 6.2(b), for a given N(t) =n, the conditional distribution of A is gamma 
a/B = 0.07 


a/B? = 0.0001, tm here, @ = 


with parameters n + a and f + £. The parameters a and f solve 


49 and B = 700. We are also given that £ = 40 and n = 4. 


The line of code given below computes the probability to be above 0.08 for a gamma distribution with 
parametersn +a =4+49=53 andf+ 6 = 40+ 700 = 740. 


pgamma (0.08, 53, 740, lower.tail=FALSE 


0.1932722 


— 


73 


CHAPTER 7 


EXERCISE 7.1. (a) We plug 2, = nd and ,, = 0 into (7.1), and note that n starts with 1 and not 0. 
The Kolmogorov forward equations become P;(t) = —AP,(t) and P;(t) = (n — 1)AP,_1(t) - 
nAP,,(t),n = 2,3,..., with the initial condition P,(0) = 1. 


(b) To show that P,(t) = e “(1 — eWay n = 1,2,..., solve the Kolmogorov equations, we write 
P,(t) = e~*, so Pi(t) = —Ae~** = —AP, (t). Also, 
Pi(t) = -Ae* (1 - eat)" +e" (n-Dde* (1 - eat =—le**(1- eat" + 
e“*(n—-1)a(e* — 14+ 1)(1 — eWatyn = —ae* (1 — eWAtyn —(n-1)a eM (1 — 
eWatyn +(n-1dae*“(1- eat? =(n-DAae*(1- eat —nde*(1- 
et)" = (n — 1)AP,_1(t) — nAP,(t). 


(c) The distribution of X(t) is geometric that models the number of trials until the first success where 


the probability of success is p = e~“*. Therefore, E(X(t)) = - = et, and Var(X(t)) = a = 
_p—At 
5 = e**(e"t ~ 1) 


(d) IfA = 4, the probability that there will be between 3 and 5 particles at week 1 is P3(1) + P,(1) + 
P.(1) = e-4(1 —e7*)?"1+e74(1 —e *)* 14 e401 — e*)5 1 = 0.051989. The mean at week 1 


is E(x(1)) = e* = 54.59815, and the standard deviation is ./Var(X(1)) = Je*(e* — 1) = 


54.09584. 


EXERCISE 7.2. (a) We plug 2, = nd and ,, = 0 into (7.1) and note that n starts with m and not 0. 
The Kolmogorov forward equations become P,,(t) = —mAP,,(t) and P;(t) = (n — 1)AP,_1(t) — 
naP,(t),n = 2,3,..., with the initial condition P,,(0) = 1. 


(b) To verify that P,(t) = Ce Je" —eAt)r-™ » =m,m + 1,..., solve the Kolmogorov 
equations, we write P,,(t) = e~™©, so Pi, (t) = —mae~™* = —mAP,,(t). Further, P/(t) = 
—mA(™*) emt(1 — esr + (P )e™ (n —m)rAe “(1 - a = —mAP,(t) + 
(ern —m)dA(e*# -1+ Ny — ear = —mAP,(t) -(n—m)AP,(t) + 

(n- m)("") I ames = eaten" = —nAP,(t) + (n-1)Aa Ca) re = 


eWay ™™ = (n — 1)AP,_1(t) — nAP,(t). 


(c) The distribution of X(t) is a negative binomial that models the number of trials until the mth 
success, where the probability of success is p = e~*". Therefore, the mean and the variance are 


E(X(t)) = = = me, and Var(X(t)) = me = me*(e-4t — 1). 


(d) Py2(2) = (erg )e OC? (1 — e-@2@))12-5 = 0.0189. The mean and standard deviations are 


E(X(2)) = (Sje@?® = 7.459123, and /Var(X(2)) = /(5)e%4(e% — 1) = 1.915354. 
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EXERCISE 7.3. (a) In the Kolmogorov forward equations (7.1), we use 2, = 0, and uw, = np, 
and the fact that the initial population size is N. We write Py (t) = —NuPy(t) and P,(t) = 
(n + 1)uP,41(t) — nuP,(t),n = 0,1,...,N — 1, with the initial condition Py (0) = 1. 


(b) The probabilities P,(t) = (“Me™ (1 —e Ht)N-n n = 0,..., N, solve the Kolmogorov forward 
equations since Py(t) = e~%** and so, Py(t) = —Nue %“* = —NuPy(t). Also, 
N N 
Pye) = —nye( )ermat(a — emtytn +(e mH — ne HE(1 — eve 


N 
= —nuP, (0) + (n+ 1)y (, 4 ) este Sew MeO Sn) uP et) ner GO) 


(c) The distribution of X(t) is binomial with parameters N and p = e“*. Therefore, E (x (t)) = Np= 
Ne~«, and Var(X(t)) = Np(1 — p) = Ne“*(1 -— ee). 


(d) P,2(3) = Gojoe C2001 — e-(0.02)(8) = 0.0437. The mean and standard deviation are 


E(X(3)) = 15 e~@°Y@ = 1412647, and /Var(X(3)) = ¥15¢e7002@1 — e- 0.02)3)) = 
0.907007. 


EXERCISE 7.4. (a) We are given that 2 = 1.3 and p = 0.2. We need to compute 


P,@) = =P) (1-27) (te) =(1-7,)(1- 5.) (SP) 


4-1 


where 


peA- Wty  0.2e0-3-92)2) — 9.2 


Po = FeUm ag = Lge ODD) — Og = 01139172. 


Thus, P,(2) = 0.060783. The mean and variance are E(X(2)) = e@-M! = e43-922) = 9.925013 


_ AtH ri _ 1.3+0.2 3-0. 3-0. os 
and Var(X(2)) = ae ut (el Bt _ 1) a oe” 3 02)(2)(eG3 0.2)(2) _ 1) — 98.76253. 


(b) Below we simulate a 50-step trajectory of the process that starts in state 1 and has parameters A = 
1.3 and p = 0.2. 


#specifying parameters 
lambda<- 1.3 

mu<- 0.2 

njumps<- 50 


defining state and time as vectors 
N<- c() 
time<- c() 


setting initial values 
N[1]<- 1 
time[1l]<- 0 


specifying seed 
set.seed (353332) 
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#simulating trajectory 
i<- 2 


repeat { 


time.birth<- (-1/ (N[i-1]*lambda) ) *log(l-runif (1) ) 
time.death<- (-1/(N[i-1]*mu) ) *log(1l-runif (1) ) 
if(time.birth < time.death | N[i-1]==0) { 


time[i]<- time[i-1] + time.birth - 0.001 
N[i]l<- N[i-1] 


1f (1==2*njumps+2) break 
else { 
time [itl]<- time[i] + 0.001 
N[itl]<- N[i] + 1 
i<- i+2 
} 
} 


if (time.death < time.birth & N[i-1]!=0) { 
time [i]<- time[i-1] + time.death - 0.001 
N[i]l<- N[i-1] 


if (1==2*njumps+2) break 

else { 
time [itl]<- time[i] + 0.001 
N[itl]<- N[i] - 1 
i<- i+2 


} 
#plotting trajectory 


plot(time, N, type="1", lty=1, lwd=2, col="blue", xlab="Time" 
panel.first=grid()) 
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EXERCISE 7.5. (a) IfA > yu, the queue will accumulate faster than customers go through the 
server, and so we expect an infinite number of customers in the system in the long run. 


(b) For A = 3 and w = 5, the long-run probability that there will be more than 2 customers in the 
a Ae 5. fAN? 
system is P(# of customers > 2) =1—P)—P, —P,;=1- (1 -4)u{1 +2+(4) | =1- 


a 1=(2)" ay3 aye 
(4=") ae (4) =(2) =0.216. 
(c) In the long run, the average number of customers in the system is 


lim £(X()) == =15. 


(d) In the long run, the proportion of customers in the system who have to wait more than | minute is 
P(T > 1) =e~%-M = e- 6-9 = 0.135335, or roughly 13.5%. 


EXERCISE 7.6. (a) Below we simulate a trajectory of a birth-and-death process with immigration 
and emigration, with parameters 2 = 1,u = 0.2,a = 0.3, and 6 = 0.1. The trajectory starts in state 10 
and ends in state 25. 


#specifying parameters 
lambda<- 1 

mu<- 0.2 

alpha<- 0.3 

beta<- 0.1 


defining state and time as vectors 
N<- c() 
time<- c() 


setting initial values 
N[1]<- 10 
time[1]<- 0 


specifying seed 
set.seed (93743765) 


simulating trajectory 
i<- 2 


repeat { 


time.birth<- (-1/ (N[i-1]*lambdatalpha) ) *log(1l-runif (1) ) 
time.death<- (-1/(N[i-1] *mutbeta) ) *log(l-runif (1) ) 
if(time.birth < time.death | N[i-1]==0) { 


time[i]<- time[i-1l] + time.birth - 0.001 
N[i]l<- N[i-1] 


if (N[i]==25) break 
else { 


time [it+l]<- time[i] + 0.001 
N[itl]<- N[i] + 1 
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i<- i+2 


if(time.death < time.birth & N[i-1]!=0) { 
time[i]<- time[i-1] + time.death - 0.001 
N[il<- N[i-1] 


if (N[i]==25) break 
else { 


time [it+l]<- time[i] + 0.001 
N[itl]<- N[i] - 1 
i<- i+2 


} 
} 
#plotting trajectory 
plot(time, N, type="1", lty=1, lwd=2, col="blue", xlab="Time", ylab="State", 
panel.first=grid()) 


wo 
GN 
oO 
N 
2 
& 
77) 
als 
° 
0.0 05 1.0 5 Bs 2.0 
Time 


(b) To simulate a trajectory with uw = 0.8, we run the same code but specify the parameter as: 
mu<- 0.8. The plot is 
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(c) In the same code we specify mu<x- 1. The simulated trajectory is 


25 


State 


Time 


(d) Now we specify mu<- 1.2. The graph is 
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(e) In the plots above we see that as the death rate increases, the population relies on immigration more 
and more. When yu = 0.2, the flock size grows from 10 to 25 birds within roughly 2 time units. 
When p = 0.8, it takes about 16 time units to grow. When p = 1, it takes about 200 time units to grow. 
When p = 1.2, it takes about 400 time units to grow. In the last two cases (u = 1 and uw = 1.2), the flock 
keeps dying out and revives due to bird immigration. 
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CHAPTER 8 


EXERCISE 8.1. (a) For each bacterium, the offspring size Z has mean w = E(Z) = (0)(0.25) + 
(1)(0.15) + (2)(0.6) = 1.35 > 1, thus the colony growth is a supercritical branching process. The 
variance of Z is 0? = Var(Z) = (0)2(0.25) + (1)7(0.15) + (2)7(0.6) — (1.35)? = 0.7275. 


The expected size of the nth generation is 100E(X,,) = 100u” = (100)(1.35)”. The variance is 


100 Var(X,) = 10002p"- 1 H” = (100) (0.7275) (1.35)"" et a 
(207.8571) (1.35)"~ 14.35)" Ay 


(b) Let 7 9<1 denote the extinction probability for descendants of one bacterium. It solves the 
equation 79 = 1 °(0.25) + 791(0.15) + (0. 6), or, equivalently, 0.6779? — 0.857) + 0.25 = 


(1) — 1)(0.6m — 0.25) = 0. Thus, my) = ~~ = 0.4167. 


(c) P(extinction of descendants of at least one of ten bacteria) = 1 — P(no extinction) = 
1 —(1 —0.4167)?° = 0.9954. 


EXERCISE 8.2. (a) Let Z denote the size of offspring. We are given that Z~Poi(A). The mean of 


Z isu = E(Z) =A. If A > 1, the process is supercritical; if A = 1, the process is critical; if 2 < 1, the 


process is subcritical. 


(b) The variance of the offspring size is 0? = Var(Z) = A. The mean of the size of the nth 


moa cs Gg recits BoA 5.4 


generation X, is E(X,,) = uw" = A”. The variance is Var(X,,) = o7u =a 


a" — ifA #1; and Var(X,) = o*n = An, ifA = 1. 


c) Let 2,<1 denote the extinction probability. It is the smallest positive solution of the equation 
0 p y p q 


(oe) 
qn (moA)" 
= nti—e4*%=e% ——— = e74 94 = eM (%-1), 
n! n! 


n=0 n=0 
Below is the code and the graph of the numeric solution 7g of this equation as a function of A > 1. 


library (rootSolve) 


lambda<- seq(1.01,8.7,0.01) 
pid<- 1:770 

for (i in 1:770) { 
equation<- function (x) 


x-exp (lambda [i] * (x-1) ) 


pi0O[i]<- uniroot.all (equation, c(0,0.99)) 
} 


plot (lambda,pi0, col="blue") 
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pid 


lambda 


. 9801000000 


0 0.9608919215 
[745] 0.0003700480 0.0003700480 
ta B eosaseeeie A eeaaGToaad [747] 0.0003700480 0.0003700480 
AO ere tsa ssaac a aee7e2s 7 [749] 0.0003700480 0.0003700480 
PARE OTeLL GR Re eee [751] 0.0003700480 0.0003700480 
flO -soHE7SieGs-0° 7047 14040e [753] 0.0003700480 0.0003700480 
Hai ease Ge osiao [755] 0.0003700480 0.0003700480 
Hie] <0. /etQe1ss G 437 e507 [757] 0.0003700480 0.0003700480 
HP] Or esdesoliee O sataiecs79 «—«S~SC~SS«EZ59] 0003700480 0.0000000000 
UHRA oR Ere nace EL [761] 0.0000000000 0.0000000000 
ido e7aeeseici a ceo4ccs7s [763] 0.0000000000 0.0000000000 
at oa ere rae [765] 0.0000000000 0.0000000000 
og cece 7G et cs [767] 0.0000000000 0.0000000000 
Peed 07 creo o orcsueer [769] 0.0000000000 0.0000000000 


EXERCISE 8.3. (a) Let Z denote the size of male offspring. It is given that P(Z = 0) = 0.4828, 

and P(Z = n) = (0.228292)(0.5586)""7, n = 1,2,.... The expected value and variance of Z are 

w= E(Z) = (0)(0.4828) + (0.228292) Diz, n(0.5586)""* = = 1.171726, and 0? = 

Var(Z) = (0)2(0.4828) + (0.228292) ¥°,n2(0.5586)"-1 — (1.171726)? = 
0.5586 2 

(0.228292) | rassepe| ~ (1.171726) = 0.45331. 

(b) The expected size and variance of the nth generation are E(X,) = w” = (1.171726)", and 

Var(X,) = o2p" 1 = (045331)G.171726)"" 1 Se 


1-1.171726 
(2.639728) (1.171726)" *((1.171726)” — 1). 


(c) Let 29 denote the probability of extinction. It is found as the smallest positive solution of the 
equation 
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My = 1p°(0.4828) + Ln-1 Mo” (0.228292)(0.5586)""1 = 0.4828 + ———= ye (0.55867)” = 
0.228292 1 = 0.228292 11 2 s : , 
0.4828 + EERE (=x = 1) = 0.4828 + jeoceneas? which is a quadratic equation 


0.5586 22 — 1.04147) + 0.4828 = 0. The solution is 7) = 0.864304. 


EXERCISE 8.4. 
(a) The mean winnings of a stake of $1 is uw = ($1)(0.3) + ($15) (0.2) + ($20)(0.1) + ($0)(0.4) = 


$5.3 > 1. Since 5.3 > 1, this the process is supercritical. 


(b) The mean winning of nth bet is uw” = ($5.3)”, and therefore, the expected winning on the fifth bet 
is uw? = ($5.3)° = $4,181.96. 


(c) Let 1 be the probability that the gambler's stake eventually turns into $0. It is the smallest 
positive solution of the equation 7) = 0.4 + 0.3 my + 0.274° + 0.172°. Solved numerically, 7) = 
0.5714957. The R script follows. 


library (rootSolve) 


equation<- function (x) 
0.4-0.7*x+0.2*x*15+0.1*x%*20 


Print (pi0<- uniroot.all (equation, c(0,0.99))) 


0.5714957 


EXERCISE 8.5. (a) The mean number of computers that are infected in one day is up = 
=(0 +1+2+3) = 1.5. Since the mean is larger than one, it is a supercritical process. 


(b) The average number and standard deviation of infected computers on day 10 are 


E(X49) = w*® = (1.5)? = 57.66504, and 


JVar(Xi9) = (- (02 + 12 + 22 +32) — (1.5)?) (1.5)9=E = /5,445.9862 = 73.7969. 


(c) Since the process of virus spread is a supercritical branching process, the virus will not die out 
with probability one. The probability of its extinction 7g is the smallest positive solution of the 


equation 7) = =(1 + 1) +12 +78), or, equivalently, 
1— 3m) +12 + 16 = (TM —1)(m — (-1 + v2)) (m — (-1-v2)) = 0. 
Hence, 1) = —1 + V2 =0.4142. 


(d) Below we simulate the number of infected computers for 10 days. A total of 424 computers 
became infected in this simulation. 


#specifying parameters 
psa "6:00:25, 0.257 0.25; 0.425) 
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N<- c() 
N[1]<- 1 


#specifying seed 
set.seed (1088878) 


#simulating offspring 
for (1 in 2:11) { 
Z<- 0 
for (j in 1:N[i-1]) { 
Z<- Z + sample(0:3, 1, prob=p) 


N[i]l<- Z 


if (N[i]==0) break 


1 2 5 9 19 28 33 52 72 90 113 


EXERCISE 8.6. (a) The code below generates the offspring of the first 20 generations of the 
branching process and calculates the total population size. 


#specifying parameters 
ps] 6 (el, O24;,- 05) 
N<- c() 

N[{1]<- 1 


#specifying seed 
set.seed (377584410) 
#simulating offspring 
for (i in 2:20) { 
Z<- 0 
for (j in 1:N[1i-1]) { 
Z<- Z2 + sample(0:2, 1, prob=p) 


if (N[i]==0) break 


N 


1 12 2 4 6 8 11 15 21 30 39 56 79 101 129 181 251 345 468 667 


In this simulation, the 20th generation size is 667 particles, which constitute the offspring of the 19th 
generation. 


sum (N) 
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2415 


There are a total of 2,415 particles in the 20 generations. 


(b) The code below simulates a sample trajectory of this process and plots the branching process for 6 


generations. 


Library (tidyverse) 


#specifying parameters 
gen.max<- 6 
p<- c(0.1, 0.4, 0.5) 


#specifying seed 
set.seed (332975) 


#simulating trajectory 
level.segment <- function(gen, y, branch.num) { 


branch<- data.frame(x=c(), y=c(), xend=c(), yend=c() ) 
gen.remaining<- gen.max-gen-1 


if (gen.remaining < 0) return (branch) 
if (branch.num > 0) { 


branch<- rbind(branch, data.frame(x=gen, y=y, xend=gen+l, yend=y), 
level.segment (gen=gentl, y=y, branch.num=sample(0:2, 1, prob=p))) 


} 


if (branch.num > 1) { 
branch<- rbind(branch, data.frame(x=gen, y=y, xend=gen+l, 
yend=y+3*gen.remaining), level.segment (gen=gentl, y=yt+3“gen.remaining, 
branch.num=sample(0:2, 1, prob=p))) 


} 


if (branch.num > 2) { 
branch<- rbind(branch, data.frame(x=gen, y=y, xend=gen+l, 
yend=y-3*gen.remaining), level.segment (gen=gentl, y=y-3*%gen.remaining, 
branch.num=sample(0:2, 1, prob=p))) 


} 


branch 


} 
bp<- level.segment(1, 0, sample(0:2, 1, prob=p) ) 
#plotting trajectory 


plot (bp[,1], bpl,2], type="n", yaxt="n", xlim=c(1,6), ylim=c(range(bp)), 
xlab="Generation", ylab="Branching process", panel.first=grid() ) 


segments (bp[,1], bp[,2], bp[,3], bpl,4], lwd=2, col="blue") 
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CHAPTER 9 


EXERCISE 9.1. 


erie es [| , ifs <t, 
(B(s), B(t)) = See @2@) __ _ mins) _ vsvt yt _ [mings.t) 
ee ~ War@G)/var(B@®)  vsvt ) sg ; aT 
Jee = [| , ift<s 


EXERCISE 9.2. (a) X(t) =tB (- ) has mean E(X(t)) = = tE (8 (- )) = 0 and variance 


Var(x (t)) =i (=) = t. Also, it has a normal distribution with independent and stationary 
increments since B(t) does. Therefore, X(t) is a standard Brownian motion. 


(b) Y(t) = a@B,(t) + V1 — a@?B,(t) has a normal distribution as a linear combination of two normally 
distributed random variables. It also has independent and stationary increments, inherited from B,(t) 


and B,(t). The mean is E(Y(t)) = @E(B,(t)) + V1 — a?E(B,(t)) = 0, and the variance is 
Var(Y(t)) = Var(@B,(t) + V1 — a?B,(t) ) = a*Var(B,(t)) + (1 — @?)Var(B2(t)) = a? t+ 
(1 — a?) t =t, thus, Y(t) is a standard Brownian motion. 


EXERCISE 9.3. 

(a) P(O< B(1) <1, 1< B(3) —B(1) < 3) = PO < B(1) < 1)P(1 < B(3) - BQ) < 3) 
(by independence of increments) 

= P(O < B(1) < 1)P(1 < B(2) < 3) (bysstationarity of increments) 

= P(0< B(1)< ee < V2B(1) < 3) (by rescaling of Brownian motion) 


= (6(1) — 0(0))(® ( ©(—)) = 0.076053. 


a) -°@) 
(b) PO < B(1) <1, 1< B(2) <3) = P(O <B(1) <1, 1—B(1) < B(2) —B(1) <3- 
B(1)) =f) Pl —x < B(2) - BQ) <3 -x | BCL) =) fey (x)dx 

= f, P(1—x < B(A) <3 —X) feqy(x)dx (by independence and stationarity of increments) 


3-Xx 1 1 
2 =e r dy dx = 0.2198108. 
“Clove? ae 


func<- function(x) (pnorm (3-x,0,1)-pnorm(1-x,0,1))*pnorm(x,0,1) 


integrate(func, 0, 1) 


0.2198108 with absolute error < 2.4e-15 


(c) P(O< B(1) <1, 0< B(2) <~) = P(O< B(1) <1, —B(1) < B(2)-— BY) < m) = 
fy P(-x < B(2) — BCA) < © | BL) =x) fac (x)dx = fy P(-x < BCL) < &) figay(x)dx 
(by independence and stationarity of increments) 
= f, P(—o0 < B(1) < xX) fea) (x) dx (by the symmetry of normal distribution) 
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2 [ cod (x) =; (©2(1) — ©2(0)) = 0.228931. 


EXERCISE 9.4. The distribution of B(s) + B(t) is normal as the sum of two normally distributed 
random variables. The mean is E(B (s) + B(t)) = E(B (s)) + E(B (t)) = 0, 

and variance is Var(B(s) + B(t)) = Var(2B(s) + B(t)—- B(s)) = Var(2B (s)) + 

Var(B (t)-B (s)) (by independence of increments) 

= 4Var(B (s)) + Var(B(t — s)) (by stationarity of increments) 

=4s+t—s=3s+t. Alternatively, Var(B(s) + B(t)) = Var(B(s)) + 2cov(B(s), B(t)) + 
Var(B(t)) =s+2min(s,t) +t =3s+t. 


uz 


EXERCISE 9.5. (a) For x > 0, P(|B(t)| <x) =P(-x < B(t) < x)= £ a e 2t du 
x/Vt 4 22 x x 1 
=2 wae” “F dz =2((—) - @(0)) =2((4)-5) =20(=)-1=F x). 
; phy x 1 V0? a 
(b) The density of M(t) is Fiyay(x) = 20' (4) =2 Fee 2 = = ez, x B 0. Therefore, 


the mean is computed as 


cio) [" [Exeriacn ten st= FE [- Fae PE 
a 0 


(c) The theoretical mean is E (M (5)) = a = 1.784124. Below we simulate 1,000 trajectories of 
a standard Brownian motion on the interval [0,5] and calculate the sample mean of the maximum 


E(M(t)). 


defining Brownian motion as matrix 
BM<- matrix(NA, nrow=5000, ncol=1000) 


specifying seed 
set.seed (8022022) 


simulating trajectories 
for (j in 1:1000) { 
BM[1,j3]<- 0 


for (i in 2:5000) 
BM[i,j]<- BM[i-1,j] + sqrt (0.001) *rnorm(1) 
} 


#computing maximum of each trajectory 
max.BM<- c() 
for (j in 1:1000) 

max.BM[j]<- max(BM[,j]) 


#computing mean of maxima 
mean (max. BM) 


1.780714 
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EXERCISE 9.6. (a) Consider the process {—B(t), t = O}. It has a normal distribution with 
independent and stationary increments since B(t) does. The mean is E (-B (t)) =-£ (B (t)) =0, 
and variance is Var(-B(t)) = (—1)?Var(B (t)) = Var(B (t)) = t. Therefore, it is a standard 
Brownian motion. 


(b) To find the cumulative distribution function of min. B(s), we write for x < 0, 
sss 


P( min B(s) < x) =. — P( min B(s) > x) = 1-P(B(s) >x, Vs € [0,t]) 


= 1-P(-B(s) < —x,Vs € [0,t]) =1 ~ P(max(—B(s)) < —x) 


=1-(20 (=) -1)=2 (1 _@ (=)) = 20 (=) (by symmetry). 


(c) P (min B(s) < -3) = 20 (=) = 0.179712. 


(d) The code below generates 1,000 trajectories of a standard Brownian motion and calculates the 
empirical probability that the minimum is less than -3 on the interval [0,5]. 


#specifying seed 
set.seed (2541165) 


#Ssimulating trajectories 
BM<- matrix(NA, nrow=500, ncol=1000) 


for (j in 1:1000) f{ 
BM[1,j3]<- 0 


for (1 in 2:500) 
BM[i,j]<- BM[i-1,3] + sqrt (0.01) *rnorm(1) 


} 


#computing indicator of minimum < -3 
ind<- c() 


for(j in 1:1000) 
ind[j]<- ifelse(min(BM[,j])< -3, 1,0) 
sum (ind) /1000 


0.179 


EXERCISE 9.7. (a) Consider the process 


t 
1-—t)B\|——], ifO<t<1, 
xo =| 1B(=—), i : 
0, ift = 1, 
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where {B(t), t = 0} is a standard Brownian motion. Note that X(0) = B(O) = 0 = X(1), so the 
process is tied at the endpoints of a unit interval. It has a normal distribution since B(. ) is normally 


distributed. Its mean is E(x(t)) =(1-t)E (8 (4))= = 0, and variance is Var (a —t)B (= -)) = 
(1 -t)? (=) = t(1 —t). The covariance is 
t t 


Cov(X(s),X(Q) = (1—s)—1t) Cov [B (=),8 (—)| = =(1—s)(1—t) min (=, i) 


1-t 1-s' 1-t 
(4-s)\(1-H)—,=s-st, ifs <t, 
= ea = min(s, t) — st. Thus, X(t) is a Brownian bridge. 
(1 —s)A-t)—= t—st, ift<s 
(b) Consider the process {B (t)=(1+ t)x (— —), t> of, where {X(t), 0 < t < 1} is a Brownian 
bridge. It is normally distributed since X(t) is. The mean is E (B (t)) =(1+t)E (x (5) = (0) 
and the variance is Var(B(t)) = (1+t)?Var (x (4))= (1+t)? (— s ——) (1 = -) = t. The 


1+t 1+t 
covariance is Cov(B(s), B(t)) = =(1+s)(1+ t)Cov(x (= —), X (4))= =(1+s)(1+1t) 


pe ea) ee | een eee 
2 (1+ s)(1 +t) min(s, t) 


1 + min(s, t) 
= min(s,t). The above properties indicate that {B(t), t = 0} is a standard Brownian motion. 


—st = (1+ max(s,t)) min(s, t) — st = min(s,t) + st — st 


(c) Let {B(t) = X(t) + tZ,0 < t < 1} where {X(t),0 < t < 1} is a Brownian bridge and Z is an 
independent standard normal random variable. The distribution of B(t) is normal being a linear 
combination of two normally distributed random variables. The mean is E (B (t)) = E(X) + 
tE(Z) = 0, and the variance is Var(B(t)) = Var(X(t) + tZ) = Var(X(t)) + t?Var(Z) = 
t(1—t)+t? =t. The covariance is Cov(B(s), B(t)) = Cov(X(s) + sZ, X(t) + tZ) 

= Cov(X(s),X(t)) + stVar(Z) = min(s,t) — st + st = min(s,t). These indicate that the 
process {B(t),0 < t < 1} is a standard Brownian motion. 


EXERCISE 9.8. (a) For0 <s<t, P(B(s) <x|B(t)=w) = J*, facsjaco(ulw) du 
= [% feosw- ByUW—w) a =", Fay) fan-B(s) (w-¥) 


FS) Bw du (by independence of increments) 


ae ees 
2(t—s) 


1 
_ Cc V27S i Ss) 
—0O 


du (by stationarity of increments) 


1 a 
Jant® e 
Be er pee 
= fe e 2s(t-s)/tdy, which means that the conditional distribution is normal with 
oo Van S(t—-s) 


mean ~ B(t) and variance (t —s). 
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(b) Brownian motion, conditional on the value of the endpoint B(t) = 0 is indeed a Brownian 
bridge on the interval [0, t]. It is normally distributed with independent and stationary increments 
(inherited from the Brownian motion). Its mean is ~B(t) = 0 and variance is “(t —s)= 


s(1-3) 


EXERCISE 9.9. (a) As seen from the picture below, P(M(T) = a, B(T) < x) = P(M(T) = 

a, B(T) = 2a — x). That is, by the reflection principle, once the Brownian motion hits level a, the 
probability to end up at time T at x or below is the same as the probability of ending up at or above 
6 aw 


2 =~ PReeeeesieeesseaeter een aart 


<0) 
T 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
See eee 


And if the Brownian motion ends up at or above 2a — x at time T, its maximum is definitely above a, 
so the event M(T) = a can be omitted. Hence, P(M(T) => a, B(T) < x) = P(B(T) = 2a — x), where 
a>Oand0<x <a. 


(b) The joint cdf of M(T) and B(T) can be obtained as follows: 


P(M(T) <a,B(T) < x) = P(B(T) < x) — P(M(T) > a, B(T) < x) = P(B(T) <x) - 


2a-X 


P(B(T) = 2a—x) = ©()-(1- 04), a>Oand0 <x <a. 


The joint density of M(T) and BCT) is found by differentiating the joint cdf with respect to a and x: 


fucr),e(r) (4, x) = __ 2 (=) 7 ( _ CF) 


2(2a—x) _(@a-x)? 
=a ° a a>0,0<x <a. 
1 


2 1 2(2a-x) ee 
=== —> 
VTV2n 9 2T 


The conditional density of M(T) given that B(T) = x is calculated as follows: 
2(2a—x) _ @a-x)* 


eT 2T 
faeeyveer (ale) = eDe@@*) _ _Tvanr _ 2(2a x) saa 
a facr)®) {22 ia 
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2(2a—x) _2a(a-x) 
ge 4 e Dm a>0,0<x<a. 


(c) Letting x = 0 in the above formula for conditional density, we obtain the density of Mgp(T), the 
maximum of a Brownian bridge on the interval [0, T]. 


4a 2a? 


fMpp (7) (@) = T e T, a> 0. 


(d) The expected value of Mgp(T) is derived as 


2a? 2a 


co co 4 
E(Mpe(T)) = | Afug,(r)(a)da = | a= e Tda={z= TP 


at 2 vt a Saget 1 |nT 
=—| ze %dz=— svi | —— e Tdz=— |—. 
25 J 2 2 La Ie 


The integral is equal to one since it is the expression for the variance of a standard normal random 
variable. 


EXERCISE 9.10. (a) The code below plots a simulated trajectory of a two-dimensional Brownian 
bridge, both coordinates of which are independent Brownian bridges on the time interval [0, T] where 
T = (24)(60) = 1440 minutes. 


#defining processes as vectors 


BMX<- c() 
BMY<- c() 
BBX<- c() 
BBY<- c() 


#specifying seed 
set.seed (6151009) 


#Simulating two trajectories of Brownian motion 


BMX[1]<- 0 
BMY [1]<- 0 


for (i in 2:1440) { 
BMX [i]<- BMX[i-1] + rnorm(1) 
BMY [i]<- BMY[i-1] + rnorm(1) 
} 


#computing two trajectories of Brownian bridge 


for (i in 1:1440) { 
BBX [i]<- BMX[i]-i/1440*BMxX [1440] 
BBY [i]<- BMY[i]-1i/1440*BMY [1440] 
} 


#plotting two-dimensional Brownian bridge 
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plot (BBX/10, BBY/10, type="1", col="green", xlab="longitude, in miles", 
ylab="latitude, in miles") 


#plotting water source in blue 
points (BBX[1], BBY[1], col="blue", pch=16, cex=2) 


install.packages ("magick") 

library (magick) 

compass<- image _read("./compass.png") 
bison<- image _read("./bison.png") 


rasterImage(compass, 0.4, 2.1, 1.4, 3.8) 
rasterImage (bison, -1.6, 0.9, -0.6, 2.1 


latitude, in miles 


longitude, in miles 


(b) From Application 9.1, we know that the expected diameter of a one-dimensional home range is 


| T | ia | T «aT 
= tenths of a mile. Thus, the expected area of a two-dimensional home range is = : = = = = 


m(1440) = 2.261.945 squared tenths of a mile or 22.62 squared miles. 


(c) The following code simulates 1,000 trajectories and computes the sample value of the area. 


#defining trajectories as matrices 

BMX<- matrix(NA, nrow=1440, ncol=1000) 
BMY<- matrix(NA, nrow=1440, ncol=1000) 
BBX<- matrix(NA, nrow=1440, ncol=1000) 
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BBY<- matrix(NA, nrow=1440, ncol=1000) 


#specifying seed 
set.seed (822815) 


#Simulating trajectories of Brownian motion 


for (j in 1:1000) { 
BMX[1,3]<- 0 
BMY [1,j3]<- 0 


for (i in 2:1440) { 
BMX[i,j]<- BMX[i-1,j] + rnorm(1) 
BMY [i,j]<- BMY[i-1,j] + rnorm(1) 
} 


#computing trajectories of Brownian bridge 


for (j in 1:1000) { 
for (i in 1:1440) { 
BBX[i,j3]<- BMX[i,j]-i/1440*BMX[1440,3] 
BBY [i,j3]<- BMY[i,j]-i/1440*BMyY[1440,]3] 
} 
} 


#computing sample ranges 
xrange<- c() 
yrange<- c() 


for (j in 1:1000) { 

xrange[j]<- max (BBX[,j])-min(BBX[,j]) 
yrange[j]<- max(BBY[,j])-min(BBY[,j]) 
} 


print (mean.area<- mean (xrange* yrange) /100) 


21.91089 


For this simulation, the sample value of the area is 21.91 squared miles. 


EXERCISE 9.11. The following code estimates parameters and simulates a Brownian motion with 
drift and volatility and plots actual and simulated bird population size against time (in months). 


(a) We plot the data first. 


birds.data<- read.csv(file="./BirdPopulation.csv", header=TRUE, sep=",") 


month<- birds.data$month 
popl<- birds.dataSpopulation 


plot(month, popl, type="1", lwd=2, cex=0, col="blue", xlab="Month", 
ylab="Bird Population") 
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(b) We calculate the increments and present the histogram. The data do look normally distributed as 
evidenced by the bell-shaped histogram. 


#computing increments of population size 
delta.popl<- popl-c(0, head(popl, -1)) 
delta.popl<- delta.popl[-1] 


library (rcompanion) 
plotNormalHistogram(delta.popl, col="light blue") 
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(c) We estimated the drift and volatility by the method of moments estimators (which are the same as 
the maximum-likelihood estimators). 
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#estimating parameters 
print (mu.hat<- mean(delta.popl1) ) 


16. 33898 
print (sigma.hat<- sd(delta.popl) ) 


29.28611 


(d) We simulate Brownian motion with drift and volatility and plot the actual and simulated trajectories 


on the same graph. 
simulating Brownian motion with drift and volatility 


specifying Brownian motion as vector 
BM<- c() 


specifying initial value 
BM[1]<- 0 


specifying seed 
set.seed (2217626) 


simulating trajectory 
for (i in 2:60) 
BM[i]<- mu.hat + BM[i-1] + sigma.hat*rnorm(1) 


#plotting actual and simulated trajectories 

plot(month, popl, type="1", lwd=2, cex=0, col="blue", xlab="Month", 
ylab="Bird Population") 

lines(month, BM, lwd=2, col="red") 


legend("bottomright", c("Actual population size", "Simulated population size"), 


lty=1, lwd=2, col=c("blue", "red") ) 
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EXERCISE 9.12. (a) Below is the script and the plot of the data. 


AQI.data<- read.csv(file="./AQI.csv", header=TRUE, sep=",") 


day<- AQI.dataSday 
AQI<- AQI.data$AQI 


plot (day, AQI, type="1", lwd=2, cex=0, col="purple", xlab="Days", ylab="AQI") 


AQI 
40 60 80 100 


20 


0 20 40 60 80 100 


Days 


We see from the graph that the values of AQI start high and then quickly decrease, suggesting that 
possibly a geometric Brownian motion might have a good fit. 


(b) The script lines below estimate the drift and volatility coefficients of the geometric Brownian 
motion model. 


#computing increments of log-AQI 
log.inc<- c() 


AQI1<- AQI[-1] 
AQI1.lag<- head(AQI, -1) 
log.ince<- log (AQI1/AQI1.1lag) 


#estimating parameters 
print (mu.hat<- mean(log.inc) ) 


-0.01867594 
print (sigma.hat<- sd(log.inc) ) 


0.1617922 
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(c) Below we plot the actual and simulated values. 


simulating geometric Brownian motion 


specifying geometric Brownian motion as vector 
GBM<- c() 


specifying initial value 
GBM[1]<- AQI[1] 


specifying seed 
set.seed (34597) 


Simulating trajectory 
for (i in 2:100) 
GBM[i]<-GBM[i-1] *exp (mu.hatt+sigma.hat*rnorm(1) ) 


#plotting actual and simulated trajectories 


plot(day, AQI, type="1", lwd=2, cex=0, col="purple", 


ylab="AQI") 


lines (day, GBM, lwd=2, col="green") 


legend("topright", c("Actual AQI", "Simulated AQI"), 


"green") ) 


AQI 
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ylim=range(GBM), xlab="Days", 


lty=1, lwd=2, col=c("purple", 


— Actual AQI 
—— Simulated AQI 
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EXERCISE 9.13. We are given that py = —0.4, o? = 0.76, X(0) = 150, K = 120, andt = 7. 


o” 0.76 
We compute r = w + > = —0.4 + —— = —0.02, A= ove 


ut-In(z5) 7 (-0.4)(7)=In(=> 


120 


a0) ae 
CE a 1.11721 and 


C = X(0)®(A + ot) — eK ®(A) = (150) ® (-1.11721 i J.76)(7)) = 


e~ © 0.02)(7)(120)@(—1.11721) = $114.21. 
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EXERCISE 9.14. (a) Foreign currency exchange rates can be modeled well with an Ornstein- 
Uhlenbeck (OU) process because if the rates are very high, demand decreases, or, similarly, if the rates 
are very low, then demand increases. In both cases, the rates will revert to some long-term mean. In 
addition, it is reasonable to assume that the variance of exchange rates stays within certain bounds 
because the variance cannot grow forever in this setting. 


(b) The code below fits an OU process to euro/US$ daily exchange rate. Plots of actual and simulated 
trajectories are also presented. 


Gl 


exchrate.data<- read.csv(file="./Foreign Exchange Rates.csv", header=TRU 
sep=" P my 


#estimating parameters 


inc<- exchrate.data$SEURO[-1]-head(exchrate.data$SEURO, -1) 
fit<- glm(inc ~ head(exchrate.dataSEURO,-1) ) 


theta.hat<- -fitScoefficients[2] 
mu.hat<- fitScoefficients[1]/theta.hat 
sigma.hat<- sigma(fit) 


simulating trajectory of OU process 


specifying seed 
set.seed (5536667) 


defining OU trajectory as vector 
OU<- c() 


specifying initial value 
OU[1]<- exchrate.data$EURO[1] 


for (i in 2:length(exchrate.dataSDATE) ) 
OU[i]<- OU[i-1]+theta.hat* (mu.hat-OU[i-1])+sigma.hat*rnorm (1) 


#plotting trajectories 
plot(as.Date(exchrate.dataSDATE), exchrate.dataSEURO, type="1", lty=1, lwd=2, 
col=3, xlab="Time", ylab="EURO/USS", first.panel=grid()) 

lines (as.Date(exchrate.dataSDATE), OU, lwd=2, col=4) 

legend("topright", c("Actual ratio", "Simulated ratio"), lty=1, lwd=2, col=3:4) 
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EXERCISE 9.15. The process {B3(t),t = 0} has a normal distribution with independent and 
stationary increments since {B,(t),t = 0} and {B,(t),t = 0} have these properties. The expected 


value and variance of B3(t) are E(B; (t)) = E(pB, (t) +./1—- p?B, (t)) = 0, and Var(B3 (t)) a 


Var(pB,(t) + 1 — p?B2(t)) = p?Var(B,(t)) + (1 — p?)Var(B2(t)) = p2t + (1 — p?)t =t. 
Thus, {B3(t),t = 0} is a standard Brownian motion. 


(b) The covariance between B,(t) and B3(t) is Cov(B,(t), B3(t)) = Cov(B, (0), pBy(t) + 
V1 —?B,(t)) = pVar(B,(t)) = pt. The correlation coefficient is 


Cov(B,(t),Bs() pt 


oe 
[Var(B,(t)) [Var(B3(t)) Vive 


(c) The increments B,(t) — B,(s) and B,(t) — B2(s) have N(0,t — s) distribution, and, hence, by 
part (a), B3(t) — B3(s) has the same distribution. By part (b), the increments B,(t) — B,(s) and 
B3(t) — B3(s) are correlated with the correlation coefficient p. 


(d) We modeled the Exxon and British Petroleum (BP) historical stock prices between 4/1/2020 and 
3/30/2021. The R code below plots the two processes and estimates the correlation coefficient. 


data<- read.csv(file="./Exxon BP stock prices.csv", header=TRUE, sep=",") 


time<- as.Date (dataSDate) 
Exxon<- data$SExxon 
BP<- dataSBP 
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plot(time, Exxon, type="1", lty=1, lwd=2, col="blue", ylim=c(0,70), xlab="Time", 
ylab="Stock prices", first.panel=grid() ) 

lines(time, BP, lwd=2, col="green") 

legend("bottomright", c("Exxon", "BP"), lty=1, lwd=2, col=c("blue", "green") ) 
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Exxon.diff<-Exxon[-1]-head (Exxon, -1) 
BP.diff<- BP[-1]-head(BP,-1) 
cor (Exxon.diff, BP.diff) 


0.8331595 


We conclude that Exxon and BP stock prices are highly correlated with the estimated correlation 
coefficient of about 0.83. 


101 


